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This single volume final report outlines the study results and design for an 
Educational Technology Assessment Model. The primary report feature is an eight 
task procedure in Section III intended to guide the assessor of a training 
innovation through the required data collection and analysis steps leading to a 
decision to accept, reject, or continue to study the innovation. Scenarios in 
Section V describe the step by step application of the ETAM procedures to the 
assessn^nt of an innovation. 
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FOREWORD 



This report presents the final results in the design of an Educational 
Technology Assessment Model (ETAM), a training management tool for assess- 
ing the long-term effects of -introducing an innovative technological change 
into the existing or planned training environment. This represents the 
final output of Phase II-A which is part of a multiphase effort called 
"Design of ^Training Systems," undertaken in consonance with the requirements 
of Advanced Development Objective 43-03X, "Education and Trafningo" 

Sincere thanks is expressed for the cooperation of the Director and members 
of the Training Analysis and Evaluation Group for making available certain 
documentation and in providing assistance in various areas of the project. 
Especially helpful was the TECEP report, which included training cost model 
algorithms, made available by TAE6. The assistaince provided by Dr^ William 
Giauque, Naval Postgraduate School, was extremely helpful in finalizing the 
d'Stision. analytic components of the procedure. Special acknowledgement is 
given Prof« Robert M. Gagne, Florida State University, who served as project 
consultant. 

The ETAM procedural approach was developed by Dr* R. B. Miller and Mr« Lo R. 
Duffy. Mr. R. E. Hallman provided management for the project; Mrs- l_. Girard, 
Mrs. IE. Latham and Mrs. S. Goodell provided secretarial and editorial serviceso 
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SECTION I 

' EDUCATIONAL TECHNOLOGY ASSESSMENT MODEL OVERVIEW 



PROBLEM 

Rapid changes are coming about in the demands put on education and training^ 
but there are also extensions being made continuously to educational and train- 
ing technology. The decision to adopt, reject, postpone, or further study a 
proposed innovation should be a rational choice based ypon the relationship of 
the costs to introduce and maintain the innovation, balanced against the range 
and magnitude of the benefits to be derived from the innovation. It is im- 
portant to identify, as precisely as possible, the range of applicability of 
the innovation in order to appreciate the economies of scale in realizing the 
benefits, whi ^ avoiding the costs of error in misapplying the innovation to 
training situations and other conditions where the innovation is irrevelant 
or contra- i ndicatq;d. d ' 

The. training enterprise is a system of interlocking activities where a given 
change may be reflected in a large pattern o€ consequences. Thus, if something 
IS made "easier to learn," the outcome can be a reduction in the aptitude re- 
quirements of the student, a reduction of training time, or a higher level of ^ 
capability of the trafned student for a given amount of training time. The 
choice of which variable to exploit as an advantage can depend on the relative 
cost advantage to the personnel subsystem in terms of available recruits, or 
to the training establishment in terms of costs per graduating student, or to 
operations in terms of skill level. The assessment technique leading to 
decisions should enable the analysis leading to these choices. 

Instructional technolo^/ embraces a wide variety of, elements. These range from 
costs and capabilities of dozens of types of vehicles, from human instructor "and 
blackboard to flight simulator. The technology also applies to instructional 
techniques and learning controls, student Incentives and motivations, transfer 
o,f training facilitations, methods of providing practice in knowledge and skill 
learning,, the organization of instructional sequence and content, and a host of 
other factors In the efficiency and effectiveness of the teaching-learning 
I process. Instructional technology also extends to the technical and administra- 
I tlve operations by which job task requirements are identified, training 
■j objectives specified, instructional vehicles are programmed for instruction, 
I and student performances measured and interpreted. All of these factors inter- 
I act with benefits and costs associated with where specific training will be 
conducted, when it will be taught, and under what conditions. The total benefit 
and -cost model to which a substantive innovation is to be applied is very complex. 

At this point the reader may have only a vague concept for the term "innovation" 
as used in the ETAM context. Early in Section III, characteristics and examples 
of training innovations are. discussed in "detail. However, here it should be 
sufficient to state that innovation is a constant or enduring change that, can 
be purposely planned far and evaluated on the basis of known criteria. Thus, 
while "change" may be the unplanned evolution from one state to another, innova- 
tion implies that the state change can 'and is planned and evaluated. 
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But the introduction of an innovation can have remote effects which cannot be 
readily anticipated. Its outcomes are only partly predictable, even by knowl- 
edgeable experts, so that the decision to accept or reject an innovation at a 
given time is based on probabilities of success and perceived magnitudes of 
risk. Rational choices should objectify, as far as possible, these uncertain-: 
ties, probabilities, and risks when the potential benefits and costs of the 
innovation are compared with alternatives. One alternative is not to make the 
change but continue as is with respect to the proposed innovation. But the 
risks of this conservative policy should also be made explicit. - 

The objective of this procedure for evaluating i^nnevations or changes proposed 
for the instructional enterprise is to formal ize'as, extensively as possible, 
the dimensions and elements of potential applicability of the innovation, and 
to structure the benefit variables and cost variables that are relevant to it. 
The outcome should be a presentation to the executive decision maker of -the 
profile of expected pattern of potential benefits and cost advantages, the 
estimated probabilities of achieving the advantages, the major liabilities that 
confront their achievement, and, where choice or rejection may not be clear-cut, 
the most critical data to next obtain for making that decision. The decision 
maker may then be in a position to evaluate a variety of innovation proposals 
that compete for an inevitably limited budget, so that maximum returns on in- 
vestment can be realized. 

A secondary and dependent objective is to develop a descriptive and analytic 
terminology that can be applied to data base descriptions of instructional 
vehicles and their relevant properties, job tasks and their important attri- 
butes, and training courses and content within training courses. By attaching 
relevant descriptors to identify the entire domain of variables in the training 
enterprise, it becomes possible to use the computer for making automatic searches 
for relevant entities in courses, and in instructional vehicles, after an Innova- 
tion has been described by appropriate descriptors from the descriptor list. 
These descriptor sets may also structure cost models for categorizing and access- 
ing empirical cost data. These objectives do not assume that the final decision 
to accept or reject will be based solely upon the items turned up by computer 
search-; rather, all ultimate decisions of this kind will be made by human experts. 

APPROACH • 

The entire ETAM procedure is aimed at supporting human judgment and reducing or 
simplifying the exercise of human judgment, but not of replacing it by automatic 
operations of selection and calculation. The large number of loose interactions 
in the development and administration of training, in the learning process, and 
between the learning process and th,e characteristics of the student, demand in- 
formed human judgmental processes for decisions and outcome monitoring. The 
procedures are intended to extend and support- human judgments by offering 
structure, reference definitions, and concepts, and in general, by trying to 
get the most use of infomation available' to the assessor, either within his 
head or external to it. 

Even where a set of automatic operations combines and integrates assessment data 
in executing the algorithms of the assessment model, the constituent data are 
also displayed to the decision maker for his inspection. Th6re is^every oppor- ' 
tunity for making commonsense checks against "black box" operations. 
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The procedural approach has also aimed at maximum consistency with Navy concepts 
and terminology, compatible with the practical requirements of making the assess- 
ment. It has been necessary, however, to formulate an organized picture of in- 
structional functions, of the learning process as related to defined task struc- 
tures, and of generalized differences in instructional requirements for different 
classes of job tasks. The intent in creating these taxonomic structures has 
been to compromise between acade:()ic precision and practical simplicity. Every 
term is defined at one or more levels in readily identified contexts. 

It must, therefore, be stipulated that the use of ETAM in the assessment of 
innovations requires persons with extensive background knowledge in Navy train- 
ing operations, who have had at least a number of days study of the ETAM pro- 
cedures and the referenced material which defines the application of the 
terminology. 

An attempt has been made to safeguard and offset both of the following types 
of errors from being grossly manifest: 

a. Accepting an innovation because its benefits have been overgeneralized, 
or its costs and liabilities have been incompletely recognized. 

b. Rejecting an innovation because the range of its benefits has been too 
narrowly perceived, or because its costs have been too narrowly prorated, 
or because its liabilities have not been presented in ways that enable 
their overcoming to be feasible. 

CONCLUSIONS ^ 

ETAM exists as a preliminary design and, as such, it has been concluded that: 

A systematic procedural methodology for assessing innovation and change 
in training can be described. 

The following of a procedural model can lead to a more effective analysis 
of a proposed innovation^ 

A decision structure can be organized that permits a systematic assess- 
ment of both taingible and intangible benefits. 

A decision sequence can be developed which makes effective use of 
efficiently gathered information. 

A taxonomic structure can be used strategically to search out the full 
range of application of any innovation or change within the instruc- 
tional enterprise. 

ETAM, as a totally manual procedure, demands an impractical amount of 
time and effort for exploring multiple alternatives, variations, and 
conditions. 

However, many components of ETAM demand human (expert) judgments which 
can be supported but not replaced with automated procedures. 
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RECOr^ENDATIONS 

It was concluded that the ETAM design provides a logical basis for evaluating 
an innovation; however, the manual aspects of the procedure cause it to be un- 
workable for anything beyond an analysis involving just a few alternatives, 
variations, and conditions. It is recommended that automated procedures be 
developed to facilitate more complex analyses and that this development effort 
be supported to include the following: 

• A study of the scaling procedures for expressing the relative "importance" 
of benefits and the probabilities of success,. as a refinement of the cost- 
benefits assessment process. 

» Development of the indexing procedures for translating the ETAM 
nomenclature into descriptions of courses, tasks, and instructional 
vehicle attributes. This would include translation rules and guide- 
lines for assigning data base search descriptors. 

• A study of the human factors requirements and the development of a 
procedural interface for an assessor to interact with a graphic 
terminal linked to a computer in performing information retrieval, 
contextual manipulation of decision data, and user data entry. 

9 The specification of an information flow model for the functions out- 
lined in the Development and Administrative Management of Training 
(Section III, Subtask 05.^4), to produce a more tightly coupled system 
for predicting training/job effects from changes in these functions. 

§ Development of a series of integrated programs, based upon the computer 
applications outlined in Section IV, which will accept user generated 
input, interact with the user to obtain refined data, and will provide 
a set of output reports to aid the decision-making process. 
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SECTION II 
REVIEW OF RELEVANT LITERATURE 



REVIEW OF THE LITERATURE ON TASK TAXONOMIES AND DECISION ANALYSIS 

The taxonomic structures offered in Task 5 in the ETAM procedures deal with 
the determination of the potential range-of-effect in a proposed innovation. 
The classifications in that section are based mainly on a sequence of develop- 
ments by R. B. Miller that bega-n in the early 1950's with his outline for the 
description and analysis of military tasks, primarily in the operation and 
maintenance of equipments. These techniques were best organized and presented 
in his 1963 article. This article also contained the classification of task 
elements or task functions which are substantially retained in ETAM. The 
structure for Stages of Learning was contained in his Handbook on Training 
and Training Equipment Design (1953), which was extended to include the con- 
cepts of Task Formats in his monograph on A Method for Determining Task 
Strategies (1974). His 1972 publication on the design of Duty Modules pro- 
poses a systematic method for the organizing of tasks into entities useful for 
training, operations, skills' descriptions, and transfer of training or assign- 
ment purposes. 

The literature on task taxonomies tends to be more extensive than illuminating, 
if one is concerned, wi th practical utility. It is outside the scope of rele- 
vance here to review the range of approaches, arguments, and findings. Miller's 
1975 paper contains such a general review with critiques. The same document 
which contains Miller's review, also contains the taxonomic positions of Gagne 
(1975) and Fleishman (1975), respectively. All three articles point to exten- 
sive reference bibliographies on taxonomic questions. 

In summary, taxonomic positions and aspirations have been based on three quite 
different starting assumptions. Miller suraiarized two of these three positions 
in 1967. One major school of taxonomic development is dedicated to the empirical 
discovery of taxonomic elements in behavior or performance. (Some theorists 
hold that behavior should be treated differently in this context 'than performance, 
others hold that the distinction is meaningless at best, misleading at worst.) 
Fleishman is the most systematic and articulate holder of this position. It is 
believed that whatever performance components are revealed, essentially by 
factor analytic studies, are structural entities (abilities) which are inde- 
pendent of the instructional technique or other influences on the learner's 
experiences which become manifest in a skill. 

According to Miller (1967, 1971A, 1971B), the second point of view is that 
behavioral "functions" are inevitably ambiguous in their definition and speci- 
fication except in the denotaBle instance; but this is not a taxonomic occa- 
sion which should be a generalization of instances. Furthermore, Miller 
insists, a large variety of quite different learning experiences may become 
associated in the expression of a skilled performance. He believes, there- 
fore, that task taxonomies are invented and have as their criterion not some 
natural structures tha-^ can be independently validated, but that their test 
is in practical utility. Thus, it is useful to distinguish between something 
called "scan and detect" and something called "interpret" because one can be 



id i 

ERIC 



ii-i 



'5 . y 



TAEG REPORT NO. 12-3 



efficiently learned by different presentations and experiences than the other^ 
and each generalizes differently than the other. Miller would say that if a 
categorization does not serve a useful design or decision purpose^ it should 
be abandoned for one that does. In this position, he treats task taxonomies 
as a branch of technology, whereas Fleishman (and many of his colleagues) 
treats task taxonomy development as a branch of science. 

Gagn^ represents a third position. In his Conditions of Learning (1970), he 
confines his interest to cognitive tasks. He identifies layers of building- 
block learning elements, where a more complex level of competence is achieved 
by putting together simpler building blocks, which are conceived as prerequi- 
sites. He thus distinguishes categories of human performance- whi ch, on the one 
hand, imply the existence of different human capabilities, and on the other, re- 
quire different sets of conditions for their learning. In later work (Gagng 
and Briggs, 1974), this point of view is expanded to encompass additional cate- 
gories of human competence including knowledge and motor skills. 

The literature on task analysis is' largely methodological and intersects little 
with theory and concepts of practice in learning and behavior organization. 
Miller (1962) has proposed a differentiation between task description, which is 
an objective statement of what a performance should do, and task analysis, which 
deals primarily with the psychological conditions for effective performance. 
In its broad application, task analyses of different kinds^support varied pur- 
poses in the personnel subsystem: job and task design, human factors engineer- 
ing, job aids, training, assessment of skills, skills identification, assign- 
ment, and selection. Task descriptions appropriate for one objective ma^ be 
inadequate for other of these- purposes . Task descriptions, as the basis for 
the development of training content, have become almost standard doctrine in 
the Navy, as well as the Air Force and Army, A variety of formats have been 
proposed, but most of them follow the structure of "stimulus situation-action- 
consequence of action" as the paradigm for the unit of description. In general, 
however ft formats for capturing problem solving activities, such as fault localiza- 
tion, have been inadequate as direct input to instructional design. Miller (1974) 
has attempted to remedy some ;of these inadequacies. 



Rundquist (1970) has laid out the most practical and comprehensive pattern of 
procedural steps known to this reviewer for describing duties and job tasks, 
and for proceeding from these descriptions to the specifications of courses 
and decailed course objectives. It is especially effective in topics about 
which many guides are deficient; i.e., the determination of the range of 
input situations and conditions with which the job incumbent must cope, and, 
therefore, to which he should be exposed, on an appropriate sampling basis, 
during training. The guidance contains many insights about the psychology 
of learning that should be applied to the instructional process. While it 
is procedurally comprehensive, both in range and depth, <f or developing and 
converting job information into training specifications and operations, the 
guide is not concerned wi th taxonomic issues, and indeed they are irrelevant 
to its [Purposes. Rundqul^t's procedures are being widely applied in the 
Navy. This job-skill oriented approach to the identification and specification 
of training objectives should facilitate the application of the ETA.M taxonomies. 
At present, this seems a reasonable conjecture rather than a confirmed 
demonstration. 
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An. excellent reference to a complete list of types and training devices is 
contained in a preliminary U. S. Navy Training Analysis and Evaluation Group 
report entitled, '*A Technique for Choosing Cost-Effecti ve Instructional Media." 
An extremely useful set of guidelines for the choice of training equipment and 
media applications is also contained in this report. 

The literature on decision theory has grown extensively over the past few years. 
A major part of this literature deals with a "(decision science" approach to 
combining subjective probabilities and multi-dimensional utility patterns. 
Schlaifer (1959). introduces some of the basic concepts for obtaining the ex- 
pected value of utility patterns. Rather separate components of the decision 
process are being combined to bring more sophisticated methods to bear upon 
the solution of complex problems. A Handbook for Decision Analysis (1973), 
developed as an outgrowth of a contract with the Office of Naval Research (ONR), 
treats a number of the major concepts and techniques in. decision analysis. It 
contains numerous examples of their application to practical problems. The 
basis for the information obtained in the examples is a consultant interacting 
with a knowledgeable person on the problem characteristics. The consultant 
elicits responses which provide the information folded into a general decision, 
framework. 

A later ONR contract on Adaptive Computer Aiding in Dynamic Decision Processes 
takes the prior work a step further by automating the" ability to track the 
responses of the decision maker in real time in order to learn the decision 
strategy, /and then to assist the decision maker in optimizing the process. This 
report contains a number of bibliographic references (many of them ONR reports). 

The estimation of subjective probabilities, in a sense, is almost a separate 
subject, and is a critical' component of the entire decision process. The 
present Educational Technology Assessment Model uses a very simplified method 
in requesting three estimates of the variable; the most likely magnitude of 
the variable, the most pessimistic estimate, and the most optimistic estimate.' 
The assumption is that almost 100% of probability i^ensity function is contained 
in the interval, and that a Beta distribution approximating the normal", results. 
Hahn and Shapiro (1967) indicate this distribution as applicable to variables 
bounded at both ^ends. The PERT methodology for estimating project completion 
times uses this distribution and, for the interested reader, is discussed in 
Battersby (1964). ETAM provides similar caveats as are contained in the ^ 
literature, cautioning the use of probabilities derived in this manner. Several 
ntethods for improving subjective estimates are discussed by Huber (1974) and 
are also contained in the Handbook for Decision Analysis . The scaling pro- 
cedures and the estimation of points on a continuum that bring about agreement 
of the model variables with reality, are crucial to the validity of model re- 
sults. Further work is proposed in this area and is outlined in Section VII 
of this report. 

I 

The economic analysis section of the model is straightforward and is based 
upon widely accepted techniques for handling investment type' decisions ^ 
SECNAV INSTRUCTION 7000. UA served as a basis for a portion of the output 
specifications for the results of the economic analysis . No further literature 
references are considered necessary here, although several are contained in the 
bibliography for the interested reader. 
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In conclusion, this overview, while brief, has maintained relevance to the 
key issues appropriate to the project. One is to establish a basis of aware- 
ness of the range of extant literature potentially relevant to the solution 
of the problems tatkled here. It seems clear that no systematic attempt has 
been previous y published to en compass /the objectives posed by projecting the 
potential implications of any kind of innovation in instructional technology 

2nrh /T''J*l°?u°^ ' ^^'^ ^""^^^^^ *° ^°P^ Wi th the search aspects of 
such a task. But the reader is pointed to the literature which may be at 
least tangential ]y re ated to this enterprise, if an inquiry to alternatives 
may.be considered desirable,. The bibliography presented at the end of this 
chapter, while small, points to extensive bibliographies contained in the 
references . ■ 
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SECTION III ' 

EDUCATIONAL TECHNOLOGY ASSESSMENT MODEL PROCEDURES 

The Educational Technology Assesspient Model (ETAM) is a series of tasks 
supported by subtasks and detailed procedures intended 'to provide guidance 
to persons assessing the value of a pro|3osed "educational innovation." 
The value will be determined by systematically considering the benefits and 
liabilities (including savings and costs), attendant risks from both internal 
and environmental influences, the potential range of applicability, and its 
worth»^in terms of certain economic measures. A major determinant of true 
value is related to the ran^e of application identified for the Innovation. 
The model o.utlines a comprehensive pattern of taxonomic elements based upon, 
psychological learning principles and instructional economics which provide 
the assessor with procedural guidance in establishing the innovation range 
of application. The general model structure, the major analytic and decision 
components, the documentation features, and. basic user guidance are outlined 
in this introductory section. 

APPLICATION AND COMMON SENSE 

The reader may be initially intimidate,d by the bulk of this manual. But, 
like an unabridged dictionary, only very limited sections may be applicable 
to a given problem. One reason for bulk is ^ts ability to respond to a very 
large variety of problems. Another reason for its bulk is that it is not 
only a structured^. classification system, it is also a condensed text de- 
scription of the technology of education, training, and learning. The text 
is essential for clarifying the concepts that give useful meanings to the 
terms used in the classification. The text itself is an instructional 
document. A technology which, in application, results in the spending of 
tens of billions of dollars a year in this country, deserves careful ex- 
position, and perhaps especially to those getting the data for making 
decisions on the funding of projects having such financial and social 
significance. 

Thfe reader who is a potential assessor of an innovation may furthermore 
be intimidated by the apparent length and complexity of procedures for 
collecting and analysing data for decision making. But there is no com- 
pulsion to follow through all stages and levels of data collection and 
analysis. The following imaginary examples fit ETAM structure. 

1. Teaching sleeping students 

Innovator: My machine can teach students while they sleep. 
Assessor: What skills can it teach? 

Innovator: Anything that can be learned by hearing spoken words. 
Assessor: How much does it cost? 
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Innovator: • Including R&D costs, procurem^t and maintenance, $20 per 
student per night, assuming 10,000 student nights per year. 



Assessor: 
Innovator: 

Assessor:. 

Innovator: 



. What's the sleep loss effect?- 

Eight hours sleep with learning becomes equivalent to seven 
hours sleep without i- 



If the student wer^ 
what he is taught i 




, how long would it take him to learn 
ght's sleep? 



Awake, it would ta^ke him ten to fifteen minutes to learn what 
he learned overnight in sleep- - - 



Decision maker (who overheard the conversation): Forget it! ^ . 

2. Brain-Coupled Learning: The Engram Transmitter . - . 

Innovator: This is an electronic device that enables me to link, by elec- 
trodes, the brain activities of a skilled performer of a task 
to the brain of a student. When the skilled performer is doing 
the job, the impulses transmit to the student who actually learns 
while the instructor performs. The student can then immediately 
perform the same thing at the same skill level as his instructor. 

Assessor: What tasks have you actually taught this way? 

Innovator: Bi cycle /iding , pursuit tracking on an oscilloscope, and celestlaV 
navigation. I also taught TV troubleshooting with the Engram 
Transmitter but it took quite a few sample exercises to do it. 

Assessor: Any difficulties in matching student and teacher? 

Innovator: Yes, great variability in both. Out of a hundred would-be 

teachers and a hundred would-be students^ only about five pairs ' 
can work our. 

Assessor: What's the saving in student training time over normal times to 
teach and learn the same tasks? 

Innovator: It ranges from 50% to 90% savings in terms of student practice 

hours. I can't explain yet why it doesn't work with all teacher- 
student pairs. You understand, the teacher is not teaching, he 
is just doing his job while his brain "teaches." 

Assessor: How many dollars and time of R&D would be needed for. production 
prototypes to be developed, with a 90% level* of successful 
matching of student and teacher? 

Innovator: I see about $50 million in R&lPand ten years of study and. 

development. We still don't really understand the phenomenon 
at work here. . 
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Decision, maker: There is half a million in my bucl^et for a high risk,^ 
high payoff venture, especially if it'>s in an entirely 
new area of exploration. Do you want to take that and 
solve the problem of matching the student and teachSr at 
levels that approximate our present aptitude requirements 
-J for the, same job? I can't use a rare-bird phenomenon, no 
• \ ^matter how good it is. 



Training in Handgun Firing 



Innovator: 



Assessor; 



I have a device and procedure to teach accuracy in flaring hand- 
guns. Controlled experiments show my method gets th.e same level 
of accuracy with 5,000 rounds fired as your training methods get 
with 7,500 rounds. I need no more instructors othan you do, and 
no more student' time than you use., Accura^^s measured by all 
your present criteria. With 7,500 roun^^'^rire^ my method, 
average accuracy zf.my trainees is SOJ^uperior to yours, plus 
or minus 10%, covering ninety percent of , the cases. Skill re- 
tention over six months without practice ^is better by 30%. 

Does your method apply to other aiming operations such as rifles, 
bazookasy^-me^tars, automatic rifles? 



Innovator: I don 



vt know if it can be adapted to them, but I don^t think so, 



Assessor: 
Innovator: 



What does your training cost? 



About 25% of 
The increase 
is a bonus. 



your savings in the cost of the ammunition alone, 
in skill and retention over^your method of training 



Decision maker: If it does what he says it Joes, let's use it. We can find 
out later if it can be adapted to other weapons. 

** * 

If one is familiar with the procedures and philosophy of ETAM, there is 
nothing inconsistent between tjiese scenarios and those procedures. The 
innovator and the assessor qui'ckly got jto'^the essential point of each 
decision: no go; study it further; ^oK^ accept. In the first example, it 
was quickly evident that the costS'^'bverwhelmed even a potential benefit, 
and in any event the benefit. w^s trivial. 

In the Engram Transmitter, the potential value of the innovation could be 
enormous as evidenced by the savings in training time, but the key risk 
factor was in its'range of applicability to pairings of teacher and student. 
No doL)bt additional studies on what could and could not be learned this way 
would alsO'-need to be done before further large commitments. In the hand- 
gun tratller, the ratio of costs to benefits is clear, and the range of 
applicability, although highly limited, is sufficiently important to justify 
an acceptance decision. 
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The ETAM procedures emphasize that investigation can be halted when .infor- 
matioJi is adequate for the human decision maker to accept, reject; op • 
study further the proposed innovation. Extensive assessment examination 
is justified- to the degree that:. * - 

1. The economic and, social commitment is very large. ■ • 

2. There is large uncertainty and' risk in the success of the pro- 
posed" innovation as -an enterprise. • • 

3. The advantage of benefits to cos-ts is marginal or dubious, ^ 
or mjiny of the benefits are intangibles. 

4. The training objecti\se is extremely important and no technical 
alternative seems to. be available. 

CHARACTERISTICS AND EXAMPLES OF TRAINING INNOVATIONS 'O ' - 

WHAT IS AN INNOVATION IN TRAINING AND EDUCATION? An innovauon is a relative, 
constant or enduring change in the procedures, objects, or functions used in ' 
any aspect of the instructional process, which may be viewed' as a benefit (or 
a liability) and has associated costs. Thus, an innovation may be a technical 
invention or extrapolation from an invention! or a structural modification to 
the setting of instruction, such. as classroom to on-the-job. 

The following are. general' classes of factors within and among which innova- 
tions can be introduced. 

1. Change in content of instruction for a given set of terminal- 
objectives. . . ' ' . 

2. Change in m edia or deyi^ce^ used as a vehicle for delivering 
instruction. 

V 

3. Modifications in delivering instructional or learning process 
functions . 

4. Shift in the environment in which a given phase .of instruction 
is given. ^~ 

.5. Shift in' the characteristics of -input stude nts to one-ljr a -set 
of courses. 

6. Shift in policy between student performance objectives , to be 
acquired at school and those acquired on the job', 

7. Change in criterion levels of performance required to complete 
a course, or to graduate frqm a school. 

8. Change in technique for present ation of conte nt for more effi- 
cient learning and tr.ansfeF oflT given class of material. 

^ 9. Modification of processes for generating training requirements 
and objectives and for converting this information into instruc- 
tional content and test criteria. ^ 

Zii III-4 
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10. Change in the organization of instructional modules as elements 
in programs of iiistructi on . 

11. Change in selecting, training, and assigTTtTi^ of instructors . 

Any list of things about which innovation is possible must be inccxiiplete. 
The preceding li^^ is no exception. ^ \ ' i 

EXAMPLES -OF TRAINING INNOVATIONS.' The following are st^ne specific examples 
of innovations. It is immaterial that. some are hypothetical proposals. 

Static Rehearsal of Complex Skills . Experienced pilots had learned in a 
simulator, space vehicle approaches and -landings under instrument and visual 
conditions. The innovation,, consisting ,pf techniques of self-imposed verbal 
rehearsals of the procedupos in these*jnaneuvers , 'shoived as much retraining 
benefit as the control subjects who ^practiced the^retr;aini,ng in a simulator. 

Systems Effectiveness Analyzer . SEA utilizes existing equtpitient, plus 
computer-based software, to enhance "existing APJRU' capabilities to evaluate 
weapon system performance. It assesses the setup and other conditions of the 
system as the human operates it procedurally, and also assesses the perfor- 
mance of the human operator by round-by-round probability of kill, and the ^ 
resulting target's probability of survival. Human performance is evaluated,, 
in the context of system performance. . , 

Scan and Detect Training . Although the radar screen is the primary target 
for this proposal, any task that emphasizes the scanning of a perceptual 
field and the detection of a class of signal within a context of clutter is 
a candidate.- . The sequence of training is as follov/s: the 'student is pre- 
sented viith the signal to be detected^in a field without background noise; 
immediately thereafter, the same signal is presented in a series of increas- 
ingly heavy background noise or disturbaace; the type of noise is varied. 
The student learns not merely to detect the signal in ideal (and unrealistic) 
conditions, but rather to read^ throu'gh noise. The. noise introduced becomes • 
greater' than encountered most'of the time, so the student becomes "over- 
trained" in reading through noise, leading to a high level of skill and 
reliable performance in reading through "moderate" noise levels. Reading 
through noise for a given class of signal in a given sensory modality,. is 
presumed to be generalizable as a skTil. The psychological principle is that 
of learning quickly to suppress perc^jDtual ground irrelevance as a "format" 
for the perceptual tasks. This is applicable to auditory as well as visual 
tasks. ' ' . . . ' 

Strident Participation in Learning Route . When a student enters- his first 
orientation course for training in a career specia-Uy^r-^be J s given a flow 
chart or critical path structure (with alternative .branch "p'oi nibs) that will 
^lead him through training to a full-fledged competence. The student can see 
where he is on this path, and what lies ahead. He can interrogate points 
along the path for one or more levels of conceptual presentation of the sub- 
ject matter and skills he is to learn — and these skills are shown manifest 
in job situations. The student, therefore, can see the outcomes of his 
learning efforts for what he has yet to learn and this, in turn, to- its 
meaning In job operations. This presentation ST'critical path in training 



.1 

III-5 



TAEG SEFC:Rr Iv). 12-3 



•• should also enable the student to access at any point, further levels of 
detail of job information (or job- irain ing infonndvicn) , either for review 
or as. "advance organizers" of future learning. The outcani; should, for 
most Jstudents, be increased motivation to learn as inanifef,t i»i (a) more 
rapid learning of course content; (b) better retention of course content over 
•periods of time; (c) better transfer of material learned in previous courses 
to n6w courses; and (d) higher perfonnance levels on perfomiance tests. 

The psychological principle ussd the rationale here is that the student 
is an active participant in his learning procei,s, has nreater mnemonic 
nchnjess while learning (with effects in learning rate', retention, and 
recall), and, ultimately, more motivation on the job. 

Predicting Student Perfonnance Levels from Types of Learning Errors . A 
technique is devised for capturing and "usTng the "lands of errors (as well as 
their i relative frequencies) made by students in the process of learning. 
Errormeasurement is begun on their first day in any class, and is cumulative 
in bui^ldir.g a profile. Some kinds of error are highly predictive of success 
in a ^iven kind of duty, depending on where in the course of learning they 
occur] Ot »rs are predictive of failure in a given kind of performance 
expecj:ation, but may predict success in other tasks or performance expecta- 
tions! Sti-11 other kinds of errors have no predictive value. Error data are' 
combined with ^learning data. 

The enror analysis technique, therefore, predicts wasnouts, and more or less 
precisely where the washout will occur in training (or on the job). Thus, it 
can prredict the student's ceiling in a class of activity. The error data also 
predict the kinds of tasks which the student may be expected to learn to do 
w^ll — thus, it is a selection procedure and may result in more precise 
student guidance. It should be emphasized that tSie appearance of large 
amounts of given error data early in learning a given -task, may be predictive 
of more eventual success c" that tasK than fewer errors of that kind. 

Error analysis depends, in part, on st)-ategy in designing tests and learning 
situations for the test. It does not add to overall training time in the 
traditional form of instruction. It does add perhaps 5% to the cost of 
preparing training^materials in terms of specialized instructor time. It 
also adds costs to the assessment process during training. 

The technique is derived from learning theory ar.d lab data that show different 
aptitudes coming into dominance at differcnc staqes of learning. But aptitude* 
interact with task content and with method of training and learning. 

Key Personnel Identific ation and Special Train^sno^, Conciderable attention 
and resource Is given in formal educaTjonTo blie slovi learner. But some 
individuals entering training are far more likely than others to become 
"key persons" within (or even outside) their trained skill specialties in 
their operating organizations, and training adapted to their discovery and 
early cultivation can have high payoff. Factors 1n addition to, or other 
than, ordinary technical skills lead to key roles. Early training may 
include identifiers for these student properties, and lead these students 
through special pathways that enhance tnt^ir ability +-.o assume key rtfles 
quickly when on the. job, rather than depend on oxterisive proving and 
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on-the-job training experiences. The key person may actually show up more 
poorly in the learning of some classroom materialr. and subtasks than many 
individuals v;ho turn out mediocre. Qn the job. 

The learning strategy is not fast-track> but a different track and style 
of instruction. It would need to be individualized or taught ttirough small 
group coaching; have minimized criteria on the "mechanics" content of train-- 
ing;.- and emphasize individual initiative in learning what is isi tuationally " 
needed; and in planning, organizing, and evaluating objectives. 

The training costs for these students will be high, and the benefits will 
not show up in standard performance tests, but only in job behaviors in the 
field. The washout and dropout rate may be high. It is estimated that 
about 5% of an ordinary class of trainees have key person potential, and out 
of every 10 key persons selected, only 8 will achieve key competence. 

General Comment . The structuring of a technology that both articulates and 
integrates functional eivcities may, in itself, be a aource of invention 
and innovation. Even though ideas may be serendipitous, they may be developed 
and tested systematically. . It also becomes feasiblt^ to fit research findings 
into application frameworks, and to extrapolate the significance of an engineer- 
ing development, such as low-cost microelectronics and display technologies, 
to instructional functions long before specific applications are designed with 
them. Thus, an accelerating tempo of cost-effective innovation can be achieved. 

For example, the adaptation of computer technologies to education and training 
is a costly undertaking and can generally be cost justified only when there is 
sufficient scal e of utilization. The application must achieve a "critica-^l 
mass" of usefuTness to a wide-range of students with a wide-range of practical 
task-learning, through a wide-range of levels of skill acquisition, before the 
payoffs can mount up dramatically. This is one example of the need for 
systematization of a technology in order to optimize the full potential range 
of resources within that technology for coping with the objectives of the 
large enterprise. 

MODEL STRUCTURE 

The model consists of eight distinct tasks to guide the assessor in creating 
a comprehensive debcription of the proposed educational innovation, in iden- 
tifying potentially more cost-effective alternatives, in determining the full 
range of application for the innovation, in assessing the risks, benefits, 
and liabilities associ^ated with the implementation and use of .the innovation, 
and in preparing a coherent presentation of the analytic results for review 
by the ultimate decision maker. The individual tasks and their interrela- 
tionships are shown in Figure III-l. The tasks are designed and sequenced 
to accomplish two primary objectives. 

0 First is tu provide a means to determine the true value of the 
innovation if it is applied to the training situation. 

Q Second is to find the full range of potential applications for 
the innovation in the training environments 
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In meeting these two objectives, the EIN^. procedures can- be envisioned as 
having'two major components. The first consists of those activities which 
permit expansion of the innovators view as to how and where the innovation 
might be applied. This type of "divergent activity^' takes place at the 
beginning of the procedural process by means of the expanded description 
of the innovation and through the range-of-ef feet analysis. Once the full 
potential for applying the innovation has been explored, the second,' or 
"convergent activity," deals with assessing qM aspects of the innovation 
across this full range of application- The second objective is to converge 
to the decision to accept orYeject the innovation. This conceptual view 
of the ETAM procedures is shown in Figure II 1-2, 

Each of the tasks comprising the Educational Technology Assessment Model is 
briefly described below: 

g Task 1 - Formalize the Description of the Innovation. The in- 
novator is provided with a numbtBr of categories for 
systematically expanding uio descriptive information 
on the innovation. The objective of the innovation, 
target applications, and the results of empirical 
J studies involving limited application bf the innovation 

are among the categories to be completed. 

« Task 2 - Develop/Examine Al t'^rnati ves to thp Innovation. In 
this task the innovator i:: requested to consider pos- 
sible alternatives to the proposed innovation which 
may require a lesser level or investment funding, and 
possibly be more cost-effective. 

9 Task 3 - Make Preliminary FeasibHity Profile. Thfe formal 

assessment procedure begins at tin's point with a con- 
sideration of potentia : 'risk^ in the implementation 
and application of the innovation, due to such factors 
as organizational iricompatibi li ties , goals/policies 
incompatibilities^ lechrolocji cal requirements, technical 
support requirements, funding constraints, and problems 
in attitudinal acceptance by users. 

9 Task 4 - Perform Analytic Feasibility Assessment. The results 
of Task 3 are analyzed in g^^eater depth, and risk re- 
duction studies and pv^ojacts are defined. Risk assess- 
ments are made in view of these studies and projects, 
and a preliminary decision treii is structured for pro- 
viding initial guidance as to whether the innovation 
should be accepted outright, accepted with the addi- 
tional expenditures for the risk reduction projects, 
or rejected, 

© Task 5 - Determine Range-of-Effect The py:^ior tasks have been 
concerned v/i th tho assessment of the innovation over a 
. limited range of application. Initially, it was with 
the target applications identih'ed by the innovator; 
in Task 4 a preliminary extended range of application 
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was considered. Task 5 contains a number of classifi- 
cation structures to which the innovation is related. 
Each structure may re:-ult in the assignment of one or 
more descriptors applicable to the innovation. These 
descriptors will serve, as search parameters in finding 
the full range of courses, jobs, or equipments to. which- 
the innovation may potentially applicable. Task 5 
also deals with describing the Denefit pattern which 
may be expected of the innovdtion when generally applied. 

• Task 6 - Perform Cost-Benefits Analysis. The decision, tree de- 

veloped in Task 4 is, refined to include more precise costs 
and savings derived from processing the tangible bene- 
fits through cost models. In addition, the intangible 
benefits are so analyzed that tney are expressed in 
"equivalent dollars,'' so that the various decision paths 
can be more easily evaluated. Equivalent dollars is a* 
utility expression rarher than a literal dollar value. 
Probabilities of implementation success and user accep- 
tance with and without rhe risk reduction projects are 
refined, and the decision variables -aw recalculated. 

• Task 7 - Perform Financial Analysis, This task is concerned 

with assessing the tangible benefits and liabilities 
(those expressable in real dollars) in terms of certain 
economic measures. The in -'estment costs and the annual 
costs and savings are calculated over a planning period 
extending a number of years into the future. Rates-of- 
return on the invested dollars a^e determined for the ' 
incremental effect of each alternative compared to 
every other alternative, alternatives consist of the 
proposed innovation^ the exi'. bing system, and any 
other approaches definod in Tack* 2 which were con- 
sidered reasonable candidates for further assessment. 
This assessment process pyx^vides a separate, distinct 
view of the value of the innovation as was gained from " 
the decision tree assessnont in Task 6. Both are in- 
, . * puts to tne decision maker. 

• Task 8 - Make Accept/Reject/Study Decision.' The decision informa- 

tion and formats have beo.n addressed throughout the 
assessment process. This xask provides guidance to the 
presenter of the analytic data to the ultimate decision 
maker. 

The model is structured so that the outputs of each task will permit a deci- 
sion as to whether the assessment process should be continued. The level of 
effort associated with the carrying out of any task should be consistent 
with the potential value of the innovation indicated at that point. 

ANALYTIC AND DECISION COMPONENTS 

The model design provides a hierarchical analytic approach and a sequential 
decision structure. The analysis proceeds froni judgmental inspection of 

iii-n 
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descriptive data to the application of^more sophisticated quantitative 
techniques intended to provide the decision maker with a more extensive 
understanding of the Innovation's potential value. Each analytic step in 
the assessment process expands the knowledge base for decision making, and 
should proyioe a higher level of confidence in any decision indicated At 
jne-same time, the process provides an Increasing awareness of the uncer- 
tainties Inherent In information to^'be presented to the decision maker, 
both the expanded knowledge base and the increased awareness of uncertainty 
become essential components of the information provided the decision maker. 
There is no requirement, however, that all of the tasks be carried out be- 
fore sufficient information is available for a decision. The, tasks are 
sequenced^so that an accept/reject decision can potentially be made at any 
point in the procedure. « ^ 

J^JtS^/^J^l^^^S: The Initial analysis centers around the formalized de- 
scription of the innovation. The objective, assumption, target applica- 
tions, and empirical data can be examined to detemine whether further 
enorts should be applied to the assessment of the proposed Innovation 
Obvious risks in the implementation and use of the innovation which might 
prevent intrinsic benefits from being realized are next considered in the 
analytic process. The opinions of experts are sought in obtaining 
estimates of success probabilities for Implementation and user acceotance 
A decision tree provides this framework. 

If unusually high levels of risk result from this preliminary feasibility 
profile, possible approaches for reducing the level of risk are then 
Identified. These will Introduce additional costs which must be con- 
sidered in the overall cost analysis. At -the same time, an expanded range 
of application for the innovatio,n is considered in assigning costs and 
savings to the decision tree. The next step in the cinalysis is to deter- 
mine more precisely where the proposed innovation might be applied. 
Taxonomic structures are used to classify the innovation so that a search 
Of various data bases containing target applications indexed with the <;ame 
taxonomic descriptors, v.'iii result in identification of a potendallv wide 
range of uses for the innovation. This analytic process provides a " 
systematic technique for generalizing the use of the innovation in order 
to complete the assessment process -in its full potential. A detailed cost 
anal/sis Is now conducted for the full range of courses, jobs, eauipments. 
f?^- ' p?.wnich the innovation might be applied. Important benefits and 
liabilities are introduced ointo^this phase of the analysis by assigning 
them equivalent dollar values. The benefits and liabilities quantified 
in this manner are combined with the actual savings and costs. Equivalent 
dollar values are identified for each possible outcome on the decision 
tree so that the analytic framework permits calculation of decision vari- 
ables which express a more comprehensive measure of the utility or value 
of a particular decision. This level of analysis is complemented with an 
financial analysis of the proposed innovation and possible alternatives. 
Measures of financial value are developed as a part of this analytic 
process. The full range of analysis discussed above has a number of 
decision points interposed in order that subsequent affalysis at more in- 
depth levels will be. justified by any decision to proceed beyond that point. 

DEpiSION FEATURES. From the preceding discussion, the sequential nature of 
the decision structure should be more or less obvious. 
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The expanded description of the innovation assists i,n deciding to continue 
or -abandon the assessment process at that point. The risk analysis 
Identifies any serious' obstacles to the successful implementation and use 
of the innovation; this occurs early in the assessment process, prior to 
the gathering of extensive range-of--effect and cost data. The decision 
tree framework forces the assessor to consider the implications'of a number 
of possible outcomes in the decision process. For example, the effects of 
successfully implementing the innovation only to have it fail to gain 
adequate user acceptance may be so serious (excessive recovery costs, in- 
adequately trained personnel, long-term potential for rejecting subsequent 
innovations which might have otherwise been successful, etc.) that, con- 
sidering the success probabilities, -the decision to accept may be judged 
too risky at this point in the evaluation. Identification of the full range 
ot application for the innovation permits decisions and planning efforts to 
be performed more realisticaily at the beginning, even though a resulting 
decision may be to implement in phases. Two independent views of the . 
innovations value are made available to the decision maker in the decision 
tree framework and in the economic analysis. The decision tr^p is refined 
to assist the decision maker to select the path with the highest payoff, 
taking into account both tangible and intangible benefits and liabilities 
The decision path resulting from this stage of analysis become^; the basis 
for the comparative analysis of the proposed innovation against the 
existing system, and also against any reasonable alternatives. The eco- 
nomic analysis satisfies the requirement of the financial component in the 
decision process by ensuring that potential alternatives for investment of 
scarce resources have been fully explored, and each has been subjected to 
evaluation to determine the respective returns on invested capital. The 
final decision documentation is organized so that the decision maker 
obtains a clear picture of tha merits and shortcominqs of the innovation, 
and also a sound understanding of the validity of the data used in the 
assessment and of the degree of uncertainty associated with the total range 
of decision information. 

MODEL DOCUMENTATION 

The entire procedural model for assessment of educational technology has 
been designed vyith several unique documentation features to enhance both 
Its usage and its maintenance. 

TASK PROCEDURAL HIERARCHY. -An overall task flow of the model has been pre-' 
sented in this section of the model documentation. Thi'^ is intended for 
those interested only in an overview the model structure. Each task in 
turn, has been documented at several levels cf detail. The top level 
consists of flows showing the subtasks to be undertaken by the assessor at 
tnat phase of the assessment. The sequential or parallel accomplishment 
ot the subtasks can be determined by referring to this level of documenta- 
u°"u ^^^^ flows show decision points internal to the task 

which direct the assessor to alternate subtasks. Following the task flow 
IS an overall task description containing a list of the subtasks within 
that task category. . Task, subtask, and page references are provided to , 
permit rapid'access to procedural information. This referencing scheme 
also permits pointing the assessor to a procedure contained within another 
tasK, thus eliminating the need for redocumenting common procedures. The 
most detailed eve of documentation is the actual procedure for accomplish- 
ing the subtask. A description section is optional in introducing the sub- 
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task procedurB which wi M ccntain one- or more steps outlining how the sub- 
task should be occompl ished. Cbnimentj are \nse>'ted with'in the procedural 
steps when it is fel t a clarification or further reference is necessary 
An optional section is availaole for a discussion of the rationale for 
undertaking the subtask. This will be used. when the reason for the pro- 
cedure may not be apparent to the ustr. • 

NUMBERING 'OF TASKS AND SUBTASKS. F.TAM consists of e:ght distinct tasks 
numbered 1 through 8. Within a task, the subusKs are numbered sequentially 
The reference to Task 5, Subtask 11, v/ould be 05.11. tach page within a 
subtask is numbered sequentially, so the reference to Task 6, Subtask 2. 
page 3, would be; 06:02,..^^( The introductory Hows and task descriptions 
. containing a list of the subtasks, are identified as Subtask 0, therefore, 
the reference to the overall flow for Task 4 would be 04.00. 

•■ The overall flows are not given a figure number; however, an individual 
f " ^ figure within the procedural sequencer, is numbered to correspond with the 

numbering of the subtask in which it appears, ^j^ierefore, the third 
'figure to appear within Task 5, Subtask 7, would be labeled Figure 6 7 3' 
All figures within a subtask procedure ar^ placed after the pmcedural steps 
Certain figures may be duplicated without the figure tium-ber designat-ion 
This is to permit the figure to be copied For use by the assessor in docu-- 
menting some asp.ect of the assessment process. 

It must be underscored, however, that the task f lows are generalized con- 
ceptual sequences. Practical factors can justify departures from the ' 
■sequence expressed in the conceptual inodel when actual assessment procedures 
are performed. !he conceptual map should not be treated as a procedural 
strai ghtjacket. 

The documentation design is "intended to allow ease in updating a"H niaintain- 
ing the document. , Changes can be made with in'nimal impact upon other 
sections As additional informatic- obtdirej r.hrough the use of the 
model, It can be inserted at tne uDproprijce poinc. 

. . USER GUIDANCE . ■ ■ 

The use^_^AW p roc^^ L^-^-'xpjH of the consultant type to re- 
-l fo'^fli^ed -descriptTon-of the umovation - and any data available 
about It - to the full range ol instructional dev;ces, courses, and course 
objectives, jobs and jod tasks, and ^..udent -.naracteri sties . The classifica- 
tion systen that guides, the assessor is <:i\omprehensi ve, procedural I'y- 
?nJ?r^rSnL'''°"S'"-' ^^'^^'^"^^^^ V H^'^^^ses c-hc! products in the economics- of 
nn L f r '"'^ r P^y^-hology of learning.^^ The an alytic tec hni ques re- 
aM^r|^de^_of^^^^^^^ tr,eot7, Ftl ttftf^l^ctH T^, 
and financia]ar^ Jn "generaT^tn-erefore , the use of ETAM 
IS not inten-pToT-iitner-the tFiiner or the training officer. However, a 
. l°^ndjjnderstan training personnel, interfacir^g- 
wiHl-irLilsessar^^eo^^ in.l^.Mliniin7Tnri^;]^ ^ 

The ETAM proccdui-t: have uesr, documen l..'.^ ac sev'eral i-vels of detail, as 
??pn'Sf ^J''"'''",'*' so tiiat, ir, i,i]-,y, the assessor will have available 
LfS^H^in fL^''''''^'' '^'^ccr.pi.sr.in^ the assessment. As competency is 
nn ^! .1? ^ f P''°c«^"'"es, Succeeding assessment exercises Jy re- 
quire reference only to the nigher level task descriptions or task flows. 
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The tasks within ETAM suggest a fom of project rrciriagement, including the 

n?fllll\t''\fl''^'-T'^ each task s comple!ed:1ts 

outputs should be retained in the project file since in manv rT^^t 
sequent tasks build upon or refine the results 0} prior tasJl 

eI;jIn%l'.th1T4d^^ 

uf bad aai:a may produce ou!puirW?.Fri-re total ly i n congruous J th soSnd ^ 
Intuitive judgment. All data estimates should hi continually, subjected tu 

Should be documented in all cases and verified for reliability 

IIl%JI"=L"°*^ *2 concerns the general application of the procedures 

to the assessment process. Each task, subtask. and procedure reprp.pntrnnfy 

;?.!PP.^f{"^^^" methodo oqy for^nducgjQHe^^ 

tions wil req uire v ariations in the apW EiFrn— fT^.s^ .p..n.oHi.irc? 

piSonapilCTraT^^ 

For e:xamp e, a general deci^i^on^ tree moderhii~b"^suggested f^FuiiThow- 
eyer as is pointed out within the text of the model, other decision tree 
structures may. needt^^e_created tv handle some types of asses m iHTTm ems 
AI.S0. the^gg^ess^ has a widel ^itude as to the d.pth to which any par?icuU; 

llcl Ifln ^. lt ' and r^soupCes available for assWsment. At 

f Lw t^- • '^JT.*^^ assessor -nould ask -the following questions : 
DO-X-"ave sufficient inf onnation to rea sonably recoim^nii rejection- of the 
innovation and halting of further analTii^T^crilTi-n-lH'i s pafti 6u l Sr step 
sSgglned?' °' °" ' ^""'^'P''^ anaT7^T?4iifTr out 1 iLd^ 

A last expTicit caveat. An assessment ttm witn a broad technical knowledge 
base and penetrating insights can use ETAM as a tool, develop a wider ?ase " 
liab miP. l^hL w^f?^ innovation, and an awareness of range of benefits and 
creai L fncn-Sh? ^11 ^ ^ men t team which lacks these capaci ties for 
aid w n Iun^« n ]\^' guarantee, for example, that a given training 

aid w 11 also be perceived by an assessor as a job aid having a performance 

fkTllU 'l'^'^^ f ' *'f^"^''^9 ™ a structure^for gild^^g 

Ihe as I^s^In! lit descriptive and evaluative informalion by 

the assessment team, but it cannot generate that inronnation. The ETAM 

nato?°of ?no'!ii-rv'^n3°HJt.^T'''' guarantee at lean a lowest common denomi- 

MgherleJeJs^^'f cStetence ''''''''''' '''''''' '''''''''^ 
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TASK SUBTASK PAGE 
" ^ / 01 '00 02 

TASK: FORMALIZE THE DESCRIPTION OF THE INNOVATION' 

It ^nn^ni?.-l;"4'^-'-''* ^ '"''"^ °^ ^^^P w^'l^ le^d to the selection Of 
. the appropriate decision fbrmat for determining the ultimate "ranqe-of-effeet 
or relevance of the innovatiN^n. The description is intended to get as much 
h.J?--f intent'and application of the innovation, as well as 
haf HpvpW^^ and empirical data, direct and indirect, that the innovator 
.hn.nn flS? Wherever the following format is incomplete, the innovator 
should improvise or add commentary. . 

oC?linrirjfi-"n!V*-T*^'"9' general, to follow the classification ' 
outline IS to simplify entering and retrieving the innovation into a data 
bank for cross-referencing purposes. 

?nffl^if """T -^^^ °f the following items should be 

. entered under the designated headings "^that are applicabll. 

I , ■ REFERENCE 

1. State/ the objectives and a summary of the Innovation. 01.01.01 

2. Descj'ibe del i very System or training vehicle. 01.02.0T 

3. Describe jobs and tasks applicable to the innovation. 01.0'3.01 

4. Desdribe relevant ;;student attributes. 01.04.01 

5. Desdribe empirical data on benefits and savings. 01.05.01 

6. Estimate, the practical importance of the training - qi 06 0-1 
product resulting from the innovation. " " 

7. Estimate resource requirements needed for further study 01 07 01 
and implementation of the innovation. • -u. 

RATIONALE \ . ' 

?^ntnrpJ^Li"^°rf available .to and used- by the innovator should be 
captured and set down. The structure .offered here will guide the assessment 
process, even though some of the information offered in this section may have 
to be modified in. the course of the assessment process. The foS? for ob- 
taining and sfetting down information . in Task 1 is a highly Lndensed versL 
ttlnnoS oTmL'°'r'''^ obtaining assessment data Even th u h 

Drocess ^n^ ^HnnHnn ^h''''°'? "^^asons . be rejected early in the assessment 
fi^?c adoption, the complete record of the innovator as represented in 

this task can be retained in archives suitably indexed for specializeS search 
and retrieval operations in the event that future developments warrant recoS- 
blsL^o^'ch^In^glnVSee^S: ''''' ''''' ^^^^"^'"^ develSpments^^^o^^^hl^^^" 
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TASK SUBTASK ' PAGE 
• .01 01 / • 01 

TASK: FORMALIZE THE DESCRIPTION OF THE INNOVATION 

SUBTASK: STATE THE OBJECTIVES AND A "SUMMARY OF THE INNOVATION 

PROCEDURE 

1. The formal description of the innovation should be preceded by a brief 
statemen.t on its major objectives. This should be'follpwed by a one or - 

JoL^^°^^'--!r^-^.°'"/5'^'"^''* }^^^ describes tha innovatfbn "irT general 
terms and its intended effect(s).. The surmiary should be completed with 
a list of keyword descriptors for describing the innovation as a re-' 
tnevable item by subject content in a data base directory. 

COMMENT: This set of descriptors may be modified .dOring and after analysis 
of the innovation's range-of-effect. - , , a aijr:>i3 



:p?r- 40 ni-,9 



TAEG REPORT NO. 12-3 . 

TASK SUDTASK PAGE 
01 -02 01 



TASK: FORMALIZE THE DESCRFPTION OF THE INNOVATION 



SUBTASK: DESCRIBE THE DELIVERY'sYSTEM OR TRAINING VEHICLE " 



DESCRIPTION . , ^ •" 

The innavation may be directed towards the functional improvement or cost 
reduction of .methods "of instructional delivery or of instructional media. 
Or it may deal with the process of developing course materials or the 
instructional process, either generally or specifically, to a course or type 
of course. Whether the effects may be direct ^or indirect, any information • 
■relevant to these factors known to the innovator should be identified here. 

' , ■ 

PROCEDURE 

1. Describe what' information is relevant to any one or more or the foil-ow- 
ing topics. ^ ' - '■ •• 

• Program or Course Design. .Does the/i'nnovati on ^deal with the 'process 
C'Of Ueveloping^indi vidual courses^ or for a program of courses leading 

to the knowledge and skill of a 'duty or rating, or^the design of an 
entire curriculum structure? The emphasis^ here is technique for the 
development and design training or 1;on^the organization and 
modularity of training cpriien-t. ; , ' r/ ^ ' 

• Instructioiial Process. Is the innovation ai,med at instructional 
technique, either through the design of improved or added functions 
in the instructional vehi.cle, or the design and presentation of 
content? Is a^ means made available for teaching^new levels or kinds 
of skills or accelerating the rate at which knowledge or skill is 
acquired or transferred to real -work- en'vi ronments'? Does-, the innova- 
tion enable different locales for instruction, to be used, or spe- 
cialization of the content of the instruction to the individual 
student and his particular needs and capabi^tties?- (This function 
will be repeated in the item below. V^This list is not\exhausti ve. 
•Where .an item is applicable, identify it by'najie and describe the 
relevant characteristics of the innovation. 

t Target Courses. Was the innovation intended to apply principally to 
one course, or related type of course? Identify the cours.e and the 
specific course content including, where possible, the name of the 
.knowledge or skill to which i.t applies. 
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TASK SUBTASK PAGE 
01 03 01 



TASK: FORMALIZE THE DESCRIPTION OF THE INNOVATION 



SUBTASK: DESCRIBE JOBS AND TASKS APPLICABLE TO THE INNOVATION 



DESCRIPTION 

In this section, try to identify: (1) the specific job or task in the opera- 
tional environment for which the innovation is intended; and (2) by implica- 
tion, the range of jobs and tasks to which the innovation applies. 

PROCEDURE 

1. Use the following terms and definitions to help identify the jobs and 
tasks to which the innovation applies. 

§ Equipment Orientation. Does the innovation apply to training in the 
operation or servicing of equipment? If so, iS it a specific item 
of equipment, or a class of equipment? Identify all particulars. 

• Work Function Dealing with Equipment. Does the innovation deal with 
the opert^tion or the maintenance or service and support tasks on the 
equipment identified in the preceding item? In whatever case, specify 
the applicable tasks within the identified work function. Where 
feasible use the Navy's Manual of Qualifications for Advancement as 
a guide for identification and terminology of specific knowledge or 
practical factors. 

§ Operational Environments. External operational environments may be 
shipboard, dock, bad weather, landing operations, etc. Internal 
■environments may be described by stress, faticjue, pain, or other 
distress. An innovation may be directed towards a class of external 
or internal environment. 

t Target Job (Rating, Billet, HBC, NOBC, etc.). Where possible, specify 
the applicable jobs targeted 5y the innovation in terms of Navy nomen- 
clature; i.e., general rapes, ratings, etc. 

- . 
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TASK SUB TASK / PAGE 
01 04 01 



TASK: FORMALIZE THE DESCRIPTION OF THE INNOVATION 



SUBTASK: DESCRIBE RELEVANT STUDENT' ATTRIBUTES 



DESCRIPTION ■ 

The innovation may be directe,d primarily to the' enhancement of desirable 
student attributes or individual differences, or to the minimizing of 
liabilities. These should be identified in this subtask. 

PROCEDURE ■ ' * ^ . . 

1. Use the foTl owing -terms and definitions to help identify. the student 
attributes that may be relevant to the innovation. Do not feeV re- 
stricted to these terms or cpncepts if others serve to clarify the. idea. 

e Prior Training or Experience. Does the innovation enhance, in train- 
ing or performance, previous training 'and experiences, or, conversely, 
IS it intended to compensate for deficiencies in previous training 
and experience? These may be dorTe by improved assessment procedures 
or training techniques or a combination of both. The training may be 
accomplished so as to facilitate it,5 transfer, plus job experience, 
to new or extended training. 

® Attitudes and Motivations. The innovation may be directed towards 
increasing the student's will to learn, interest, attenti veness , and 
participation in the learning process. These factors should be 
differentiated from those that may be successful in recruiting candi- 
uates for a given course of instruction: It should be understood 
that the normal by-products of effective instruction are positive 
student attitudes and motivations. An innovation may.be conceived, 
however, that seeks motivational enhancement by specific planned 
actions for that particular purpose," al though they may be embedded 
in normal instructional actions. 

• Aptitudes. The target of the innovation may primarily be that of 
changing aptitude levels or mixes while holding the objectives of 
training and training costs constant. Such targets can be identi- 
fied, named, and usually quantified. 

« Social Factors. Social environments of some students may create- 
impediments to the learning process and the learned product. They 
may also affect the social climate in the training environment and 
later the operational environment. An innovation may be aimed at 
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01 



04 



02 



-• overcoming general or specific social handicaps of entrants to train 
ing, or students, with the objective of changing acceptance patterns 
of candidates to training and, both efficiency and effectiveness of 
the training administered. 

• Washouts and Dropouts. The innovation may aim at the reclamation of 
actual and potential washouts and dropouts from training programs. ' 

• Student Evaluation. Procedures for capturing, organizing,' inter- 
preting, and using .various means of assessing student performan&e, 
during and at the end of training, for purposes other than directly 
concerned with the training process itself. 
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TASK; FORMALIZE THE DESCRIPTION OF THE INNOVATION 



SUBTASK: DESCRIBE EMPIRICAL ON BENEFITS AND'SAVINGS 



DESCRIPTION 

The innovation may have been subjected to pilot testing with live subjects 
under experimental or demonstration conditions. Presumably, the study has 
been written according to the usual requirements for replicability of what 
was done. All reports dealing with data obtained from the innovation in any 
kind of use or investigatory tryout should.be included as part of this sub- 
task documentation. 

If the innovation is directed principally to a cost reduction iii a type of 
instructional vehicle or medium, or to an instructional function, cost data 
based on eventual procurement costs, plus operation and jpaintenance costs, 
should be obtained as part of the present documentation. 

PROCEDURE ■ - ' . 

1. Obtain and set down data from the studies that show quantitative benefits 
in the learning process, or products of learning as. characterized in 
Subtasks 01.02, 01.03, and 01.04. Note any qualifications that should be 
applied to these benefits such as the distributions of scores around 
mea.n results, special conditions under which the study was performed, 
size and nature of the sample uf subjects, and so on. These benefits 
should be compared with respect to control conditions that represent a 
current practice, or that can be related to a current training practice. 
The generalizations made in the conclusions reported about the study 
should be carefully checked against the actual conditions and samplings 
in the study. 

• Criterion Varia"ETes for Measuring Training Effectiveness. If the 
innovation is directed primarily to a student performance benefit, 
the experimental conditions may enable the benefit to be characterized 
by one or more of the following. 

a. Parsimony in what is learned, with respect to a given level and 
range of acceptable task performance, or- its equivalent 1,n the 
expression of "knowledge." This is the minimulm range and level 
of learning that enables reliability and competence in all 
criteria of effective job task performance. Thus, it is one 
measure of the efficiency of training; only that which contributes 
to a performance criterion is learned. 
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^•/Validity and completeness in level of proficiency. The innovation 
/ may be directed towards increasing the job task relevance of what 
r IS learned, or to higher levels of proficiency than is practicable 
\ wi.th current training practices. 

c. Neompleteness in range of what is learned. The innovation may 

enable kinds of skill to be acquired that heretofore were not 
feasible other than in operational situations; or the innovation 
may identify skill requirements heretofore neglected in training, 
but which contribute to overall job effectiveness. 

d. Extensibility of skill in degree or in kind. If the innovation 
enables the student to increase his skill level more effectively 
by on-the-job experiences as a function of the kind of training 
he gets, the innovation would be classified under this rubric 
rather than b. above. The innovation may, however, demonstrably 
facilitate the acquisition of associated skills relative to the 
incumbent's assigned job, or related jobs, as in cross-traininq, 
or for a higher level job. 

e. Work motivation. The innovatiqn may have a primary thrust in the 
increase of the student's motivation to do the work, do it well, 
or be willing to perform effectively in potentially incapacitating 
stress conditions. This motivation should be differentiated from 
willingness to accept a given billet or job positi/on. 

2. Obtain and set down the quantitative costs reported in the studies (or 
based on further inquiry) as measured~B7The relevant input conditions. 
Costs may be expressed in comparative practice time or exposure time, or 
in dollar cost per unit of learning per student if the innovation is 
primarily an instructional vehicle function. In the latter case, determine 
or estimate the eventual unit procurement cost plus estimated costs of 
operation and maintenance (per year) of the device or medium. Set down 
any factors that may qualify these estimates. 

3. Estimate the magnitudes of benefits and savings under actual conditions 
in the Navy context. This is the translation of the findings in the 
study to values in benefits and savings, or both, in actual training 
environments and performance environments. This translation is a judg- 
mental process. The rationale for extrapolating magnitudes in kind and 
amount from the study to real life should be made explicit. The data 
derived in this step should be identified and differentiated from the 
data reported in step 2. 
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TASK: FORMALIZE THE DESCRIPTION OF THE INNOVATION 

SUBTASK: ESTIMATE THE PRACTICAL IMPORTANCE OF THE TRAINING PRODUCT 
RESULTING FROM THE INNOVATION . ^ 



DESCRIPTION 

The innovator may have performed or referenced inquiry into the importance of 
the instructional objectives towards which the innovation is directed Such 
information can assist in evaluating the potential contribution of the 
innovation. 

PROCEDURE 

1. Describe or estimate the practical importance of the training product whi^h 
results from the effects of the innovation. This may be a gross estimate 
or the importance and central ity of the knowledge or skill to effective 
job performance as a contribution to the success of the Navy mission. 

The training outcome to which the innovation is relevant may be critical 
important, potentially useful, or irrelevant to mission effectiveness •■ 
that IS, in the mainstream of job actions such as landing an aircraft, or 
peripheral to the mainstream such as the legibility of handwriting in 
filing a flight report. If actual on the job data are available (such as 
accident reports, specific kinds of failures in diagnosti c "operations , 
procedural inadequacies, and so on) that can focus and quantify the defi- 
ciency towards which the innovation is aimed, such data can materially 
assist the evaluation. 

2. The empirical benefits plus the practical importance of the innovation may 
be sufficiently clear-cut to justify going directly to Task 6 (Cost-Benefits 
Analysis), and an acceptance decision. 
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TASK: FORMALIZE THE DESCRIPTION OF THE INNOVATION 

\ 

SUBTASK: ESTIMATE RESOURCE REQUIREMENTS NEEDED FOR FURTHER STUDY AND 
IMPLEMENTATION OF THE INNOVATION 



DESCRIPTION 



Iri?w??f/ ""^ performed on the assumption that further research and development 

n ^c??L \caT\'n 'n innovation. to a state of implementation 

ur) ^ practical scale and in real training s tuations. It mav also Hp ;,qcnmpH 
as background for this step that further'testing o? the efSt ieness of the 

r?on ]??ons"a d'^^l'f '""^'"J^ ''''''' ^'"^'"9^ ^'^^ vanTus aSple ' 
OT condit ons and subjects, may be necessary or at least desirable for makina 
firm commitment about the benefits and range of benefits to be d riveS ?rom 
the innovation, as well as to expose its liabilities. 



PROCEDURE 



^" ZV.itJ breakdown estimate in, dollars, time, and human talent of the 
?L effectiveness developing the innovation and in testing 

^■■"himan'Jalerltlor:''^'''""' ^'^ 

a. Further research and development effort required to bring the innova- 
tion to level of development where it can be introduced and used 
effectively on the scale and in the contexts of practical application. 

tnSf^'^^''^"'* ^f^l^'J empirical data collection on the magni- 
tude and range of effectiveness of the innovation that can assure 
Detter than a 90% probabi 1 i ty^that the projected benefits will be 
achieved and the projected liabilities kept within their estimated 
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TASK; DEVEWP/EXAMINE ALTERNATIVES TO THE INNOVATION 



2.01 




CONSULT 
SOURCES FOR 
DEVELOPING 
ALTERNATIVES 



2.02 
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DESCRIBE 

ALTERNATIVES 



2.03 





MAKE 
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ASSESSMENT OF 
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TASK: DEVELOP/EXAMINE ALTERNATIVES TO THE INNOVATION 

The first embodiment of an innovation, may incorporate the concept but lack 
implementation know-how. A -few hours of directed effort in a selected group 
of experts can result in alternative embodiments that may sacrifice little of 
the benefits of the innovation but achieve them at costs greatly below those 
implied by the original embodiment. It may also \urn out that the innovation 
is not an innovation in fact but only in its form of presentation, and that 
the concept has been previously rejected in fact and in principle, or that it 
is embodied in existing practices but with a different name and appearance. 
The search for alternative embodiments does imply that the functional principle 
and objectives contained in the Innovative concept are understood and transmitted. 

Alternatives may also be found by searching the literature or by redirecting 
the expertise of the innovator. 

General knowledge in the subject field, plus common sense, may direct efforts 
at gross cost reduction on the more cost-sensitive variables. A cost-sensitive 
variable is one that has a large multiplier factor; it can also be a cost 
factor with many cost dependencies. 

The level of human expertise required to effectively implement the innovation 
in training practices can be a. crucial cost factor, not readily translated into 
dollars but central to the practicality of the innovation. This is especially 
true if any exercise of the innovation in the real world depends on a high level 
of creative or logical talent. 

On the other hand, an innovation that may be impractical at one time or in one 
context may be highly serviceable in other contexts. Failure history should, 
therefore, be examined for its relevance to current or projected needs and 
opportunities. For example, a simulation function that was impractical for 
.analog computers may be trivial in cost on a contemporary digital computer. 

A commonsense search for functional alternatives and their evaluation may save 
extensive time and effort in putting the original statement of the innovation 
through the assessment mill. A potential source of alternatives can be the 
archives of innovations that were proposed in the past but rejected for various 
reasons applicable at the time but no longer applicable. 

Any alternative to the innovation as it was originally proposed should be 
described in the same format that was used to describe the original proposal 
(Task 1). 
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SUBTASK 

1. Consult sources for developing alternatives to the 
innovation. 

2. Develop and describe alternatives to the innovation 

3. Make an intuitive assessment of the alternative. 



REFERENCE 
02.01 .01 

02.02.01 
02.03.01 
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TASK: DEVELOP/EXAMINE ALTERNATIVES TO THE INNOVATION ' 

^ ■ 

SUBTASK: CONSULT SOURCES FOR DEVELOPING ALTERNATIVES TO THE INNOVATION 



PROCEDURE 



Consult available sources of information and i.deas' leading to^ the develop- 
ment of alternatives to the innovation that may be potentially more feasible, 
practical, or otherwise have more benefits and fewer liabilities than the 
innovation as proposed. • ' 

Potential sources for alternatives may be found in the following ways : 

• Understanding the objectives of the innovation so clearly that some - 
aspects of the innovator's implementation become recognized as non-- 
essential or irrelevant to the central objective of the innovation. 
Stripping off the appurtenances from the innovation as originally 
pV'oposed may make it more practical and feasible to accomplish. 

• Reviewing the literature on the topic with which the innovation deals. 

. Alternatives may already exist, and data may be available about them. • 

o 

• Finding experts on the topic and consulting with them, getting their 
suggestions about alternatives to the innovation as proposed. Experts' 
may also be used to make preliminary reviews and technical assessments 
of alternatives to the innovation when these alternatives are' develope'd. 
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TASK: DEVELOP/EXAMINE ALTERNATIVES TO THE INNOVATION 
SUBTASK: DEVELOP AND DESCRIBE ALTERNATIVES TO THE INNOVATION 



DESCRIPTION 



™oE °n "° or time-consuming than is dictated by 

SMStoHTH'S^^^-"* alternatives. "Some ideas as alternatives can be 

evaluated. and dismissed in the course of casual conversation. Others may 
justify no more than a few notes of description that may be sufficient to 
In .i! "^^y weakness that eliminates the alternative from further examination. 
An alternative that survives the incubation and early critical processes, 
should be set down in sufficient detail so that if it fails in later stages 
or even informal assessment, it can be put into the data base of "innovations. 

PROCEDURE . s - • ,-^5^ 

1. ' Prepare a . description of the alternative to the inno^vation as proposed 
• by the innovator. The description should be fomatted according to the 
. outline in Task 1.. Go no further in detail than seems justified by the 
apparent quality of the alternative. - . 

2-. If. as,a consequence of interplay between the inventive, descriptive, and 
critical^ processes in the development of the alternative, it be?Jmes a 
promising candidate for competing with the innovation 'as proposed, a formal 
description should bemade of it. Follow the format described in Task 1 
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02 03, 01 

TASK: DEVELOP/EXAMINE ALTERNATIVES TO THE .INNOVATION ,• " 

■ ! 

SUBTASK: make AN INTUITIVE ASSESSMENT OF THE ALTERNATIVE 
DESCRIPTION • - ~' 

The formal description of the alternative may mt% apparent some 'difficaUip.; 
^ ^blpr^h°^''"'' befocethat description. The fonnal descr ption al o 
a t1 nat ve as'the^H.-SV^ associates to back off from the statement of the 
alternative as they did from the original description of the innovation bv thP 
^innovator. The question can be asked, globally and analytica T 'ooes this 

PROCEDU'!?^ 



Ji\i: 



1. Make an intuitive -assessment of the description of the alternative. 

2. If the result of the intuitive assessment is favorable, 'complete the formal 
description of the alternative. It should then be put through L etAM ^- 

V.'^fZ^^- T-^^^' Yl'' ^ through, 8. If the result is unfavorable, put 
the description'and the assessment. into the archives. '^'^'^^'e, put 

IIIh Jnn^^f-^^'v^ *° thje original innovation may not have equivalent objectives 
' .n.i^c?c*'°!l^^ capabilities. Jf so, the alternative must have its own 
analysis made, not only through- Tasks 3 and 4, but also through the rLe- 
of-effect .and benefits/liabilities assessments in Task 5. It is Lecessarv 

for' it%nV'p'J\'^-'%'? Objectives equivalent to the in ov io in rdeT 
' for It to be subject to comparative analysis. 
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TASK 




ESTIMATE 
IMPORTANCE TO 
PROJECTED 
NAVY MISSION 



3.02 



PSTIMATE 

ORGANIZATIONAL 

COMPATABILITY 



3 .03\ 



ESTir 

G0ALS/P5I 
CPMPATABILITY 



3.04 



ESTIMATE 

STATEOF-THE 

ART 



3.C6 



ESTIMATE 
R&D FUNDING 



3.06 



ESTIMATE 
REQUIRED'' 
TECHNICAL 
SUPPORT 



3.07 



ESTIMATE 

ATTITUDINAL 

ACCEPTANCE 



3.08 



SUMMARIZE 
PRELIMINARY 
FEASIBiLrTY 
PROFILE DATA 



rWE 
ls/pSittcy 
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TASK: -MAKE PRELIMINARY FEASIBILITY PROFILE 

judgments. Task 3^' g^' s he rre^?o^ 'a olc nVlh'' '''''' "lacro-type 

IS^wrcoM:xL^:"^hrfi?:? iT^Zr^rr'V'''''''r Alternatives' 
training perfonnance! d-crea ed "oursTlenoJh I'l^^'^V' ^f'^^"'-^- ^'"P^^^ed 
examples of intrinsic benefits rp1nl??nn J^ ' "^^^^^ed costs, etc.. are typical ^ 

for viewing these ?wo assess' n^?'"! assessment process.. A framework 

tree in Figure 3 0 1 The nt?L?c bPnPfS" '''^'''''^ ^° ^^^^ decision 

important outcomes A throuah n ihp ! ''^ aggregated for each of the 

to be considered here and in Ta;k I J^,^,°lf ^'^''^ the ones 

bin iti-es-at points u thJouaK 7 ri' " ?i to -establish the proba- 



SUBTASK 
1 



REFERENCE 

iT.?nTVl ^"novation to the Projected Navy 03 01 01 

butfnn ;f ?h^'-' 3"°?' ^'}™^^^ °f the potential ^ontri- °^ 
bution of the innovation (if successfully implemented) 
to the support of specific, identified elemen s in the 
Navy mission. These elements may lie wi thTn tf e Jrainina 
mission in the operational mission, or in a relationshiD 
. .between the training and operational mis.sions 

^' iTh'^h* r^f^Patability. Estimate the deqree to 03 0? m 

which the effects of the innovation fit in with command 
and career organizational structures of jobs and 
and policies for the use of personnel. 

^* Compatability. Estimate the degree to which 03 03 01 

the proposed innovation fits with existing goals and °^ 
policies within the training and other personnel o^qani/a 
tions wnere implementation is expected ?o take place 
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TASK SUBTASK PAGE 
03 00 03 



State-of-the-Art. Make an estimate of the technical 
feasibility of implementing the proposed innovation on 
the scale necessary for practical application. Consider 
any unique technical requirements that need extensive 
R&D effort to accomplish. 

Funding. Make an estimate of the magnitude of dollar 
support fcr further R&D effort in bringing the proposed 
innovation to the point of actual operational introduction 
and application. These costs should include those to be 
spent in modifying existing facilities and practices in 
order to accept the innovation, and costs from rptvaininq 
persons required to implement the innovation. 

Technical Support. Estimate the extent to which qualified 
personnel of whatever kinds are needed can be made avaH- ' 
able to support further R&D or implementation of the innova- 
tion or maintenance of the innovation in practice High 
levels of specialized, generalized, or creative talent 
(especially the latter) are. hard to find, and many demands 
compete for their services. 

Attitudinal Acceptance. Estimate the degree to which the 
instructional community and/or the students will accept 
the device or function as desirable, meaningful, and 
appropriate to their sense of role and self-image. 

Summarize the Preliminary Feasibility Profile Data. 
Identify risk by category, numbers of risk reduction 
projects, and estimated costs for these projects 



REFERENCE 
03.04.01 



03.05.01 



03.06.01 



03.07.01 



03.08.01 



RATIONALE 

I^t^.^n^f i^'"' '^'9'' ^° '"^"^ impediments that the 

asses or feels Justified in recommending a rejection of the proposal to the 
decision maker without further analysis. If this is not the case thP d^ta 
deve oped here will feed into Task 4 (Perfoni, Anal^ic Feas bi S essment) 
Rejec?/lL;f SeI?s?on)!'"'°"^ Cost-Benefits Analysis) and Task 8'(Make Accepi/ 



COMMENT 



Thp assessor IS cautioned here on a decision to reject without an explicit or 
intuitive understanding of the range of applications for the innovation. Be- 
cause virtually every major innovation at its inception has presented almost " 
insuperable obstacles, shortsightedness or conservatism at this pSint m y" cause 
premature rejection of a potentially valuable enhancement to training. ^ 
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TASK: MAKE PRELIMINARY FEASIBILITY PROFILE 



SUBTASK: ESTIMATE THE IMPORTANCE OF THE INNOVATION TO THE PROJECTED NAVY 
MISSION 



PROCEDURE 



1. Estimate the overall significance of the innovation and its implications 
for the goals and mission of the Navy. These may consist of the mission 
ot training, operations, or of the relationship between the two. The 
estimate should take into account future trends and demands and restric- 
tions that may be placed on Navy institutions in terms of student and 
instructor input, turnaround time in preparing and administering training, 
flexibility, and similar issues. The potential cost advantages of the 
innovation should be treated as secondary in arriving at a judgment of the 

' functional importance of the innovation. 

2. ,.Select the" statement below which best describes the estimated importance 

ot the innovation if it were implemented and widely applied to its 
potential range of usefulness. 

( ) The innovation would have an effect of little or trivial impor- 
tance to the overall training and operational missions of the 
Navy. 

( ) It would have an effect of moderate importance. 

( ) Its effect would be substantial and important. 

( ) Its effect would be highly important and possibly crucial to 
training and/or operational missions of the Navy. 

3. Decide if the innovation has iinportant enough application within tha Navy 
to justify pursuing the analysis. This consideration should be extended 
into future years where the possibility of changing missions may change 
the importance estimate. ' j ^ 
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TASK: MAKE PRELIMINARY FEASIBILITY PROFILE 



SUBTASK: ESTIMATE OF ORGANIZATIONAL COMPATABILITY 



PROCEDURE 

1. Consider the degree to which the effects that may be produced by the 
innovation, or that may be involved in its use, conflict with or fit 
into command and career organizational structures of jobs and duties 
and policies for the use of personnel. This judgment should not be 
affected by whether the assessor believes the outcome from the innova- 
tion would be desirable or undesirable. 

2. Select the statement below which best fits. The innovation would lead 
to consequences that would be: 

( ) Incompatible and inconsistent with Navy coimiand and career 
structures as they exist or are planned. 

( ) Somewhat inconsistent with Navy conuiand and career structures. 

( ) Irrelevant to Navy comnand and career structures. 

( ) Supportive of Navy command and career structures. 

3. Decide whether the risks are so high that no viable set of implementation 
projects could be described to reduce them to an acceptable level. If 
this is the ease, then rejection should be considered at this point. If 
potentially high risks exist, then thought should be qivpn at this time to 
risk reduction projects to be detailed in Task 4. If'the projected project 
cost seems unreasonably high in relation to the benefits associated with 
the reduction of risk, then this is perhaps another reason for rejection 

at this time. 



TASK SUBTASK PAGE 
03 02 01 
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TASK: MAKE PRELIMINARY FEASIBILITY PROFILE 



SUBTASK: ESTIMATE OF GOALS/POLICY COMPATIBILITY 



PROCEDURE 

1. Consider the degree to which the proposed innovation fits with existing 
goals and policies within the training and other personnel organizations, 
where implementation is expected to take place. Thus, for example, the 
goal of the training establishment could be the training of the person 
not only for a specific military job or position, but for any of a 
family of jobs, or for a counterpart in the civilian specialty as well 

as the military job. It is assumed that the assessor is familiar with 
those goals and policies that could be applicable to the innovation. 

2. Select the statement below which best fits the case. The innovation 
would lead to: • 

( ) Substantial conflicts with existing goals and policies of one 
or more training and other personnel organizations. 

( ) Minor conflicts with those policies. 

( ) No conflicts. 

( ) Support, in general, of existing goals and policies. 

3. As in the preceding procedure, an assessment of risks, risk reduction 
projects, and the apparent cost-benefits of these projects should be 
made to determine if the innovation should be rejected at this time. 
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03 04 01 

TASK; MAKE PRELIMINARY FEASIBILITY PROFILE 

SUBTASK; ESTIMATE OF THE STATE-OF-THE-ART APPLICABLE TO THE INNOVATION 



Don, 



CEDURE 



n? t^p cr^ ^^^^^"^cal feasibility of the proposed Innovation In terms 
of the scope and scale that would make Its implementation practical. 
litl ?f ""'''"^ technical requirements that might need exten- 
sive R4D effort to accomplish, In order to develop and Integrate the 
Innovation into an Instructional or operational system. 

2. Select the statement below which best fits the case. The innovation has; 

( ) Low probability of success 1n overcoming- technical difficulties, 
even with substantial continued R£D effort. 

• ^ ^ oSn^'^^^ probability of technical success with some continued 
KaU effort. 

( ) High probability of success with at least moderate ,R&D support. 
( ) Amply demonstrated technical feasibility at this time. 

^' S^SsoMc''^^^?!; J^V^^'^'u'i:^ '° ^'^^^ "° set of implementation 

projects could be described to reduce them to an acceptable Irvei. If 

II 11 u^^u* rejection should be considered at this point, if 
poten.1 ally high risks exist, then thought s.hbuld be given at this time' to 
risk reduction projects to be detailed 1n Task 4. If the projected pl^ie?t 
cost seOTs unreasonably high 1n relation to the benefits associated with 
at th^ ?r°" then this 1s perhaps another reason for reject on 
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03 



05 



01 



TASK: MAKE PRELIMINARY FEASIBILITY PROFILE 



SUBTASK: ESTIMATE FURTHER R&D FUNDING 



PROCEDURE 

1. Examine the number and kinds of R&D effort still required to be completed 
in otder to bring the innovation to completion that is, to actual opera- 
tional introduction and appTication. The R&D may be in terms of systems 
design, hardware, software, communications, and other factors . (See Task 5, 
Determine Range-of-£ffect , Subtask 2, Develop Background for Instructional 
Vehicle Analysis, for a list of potentially relevant usage properties.) 

The costs of technical and operational debugging of new technologies should 
not be underestimated. Costs required to modify existing facilities and 
practices in order to accept the innovation should be included, as well as 
costs from retraining persons required to implement the innovation. It may 
be sufficient to make gross estimates of potential costs for various 
elements of the expected R&D effort. 

2. Estimate the relative magnitude of the expected R&D costs. This estimate 
should reference the available or expected overall R&D budget of the Navy 
for training. Select the -statement which best fits the case* 

If a comparison is made with the typical size of funding for an R&D 
objective in Navy education and training, the R&D expenditure 
realistically to be expected in order to bring the proposed innova- 
tion into operational implementation will be: 

( ) Large to very large to bring the innovation to an operational 
level of implementation. 

( ) Moderate. 

( ) Readily integrated into existing planned programs of expenditures 
for training R&D. 



Decide whether the costs are so high to preclude obtaining adequate funding. 
Alternative approaches at a lesser level of funding should be considered at 
this point. If potentially high risks exist, then thought should be given 
at this time to possible rejection. If the projected R&D cost seems un- 
reasonably high in relation to the benefits associated with the reduction 
of risk, then this is perhaps another reason for rejection at this time* . 



) An immediate and relatively direct savings to the 
training. 



R&D budget for 
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Retain any estimates of actual dollar costs that were developed in this 
subtask. Document these d^ata in the project file. 
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TASK: MAKE PRELIMINARY FEASIBILITY PROFILE 



SUBTASK: ESTIMATE REQUIRED TECHNICAL SUPPORT 



PROCED'JRE . ■ 

1. Examine the various kinds of skills and abilities that will be needed in 
order to bring the proposed innovation into, service. These may require 
those skills needed for the R&D work and also those skills that will be 
necessary to maintain and apply the innovation'/ assuming it has been 
implemented. Then, assess the ease or difficulty in finding people with 
these s.kills and skill levels in the quantity that may be needed. It 
should be recognized that novel creative skills are always in short 
supply and heavy demand, especially when various technical constraints 
may be placed upon their talent. 

2. Select the statement below which best fits the estimate of level of 
technical support that will be required to develop and maintain, in 
practice, the innovation: 

( ) Extremely rsre kinds of talent will be required to (1) continue 
the R&D or; (2) to implement the innovation in practice orj 
both (1) and (2) apply. 

( ) Some difficulty should be expected in finding the kind of 
talent needed to, complete the R&D work and to implement 
the innovation in practice. 

( ) Adequate talent is available or can be made available for 
bringing the innovation to practical and realistic imple- 
mentation in training. 

{ ) Adopting the innovation will actually reduce the level of 
talent now required for performing the function intended 
for the innovation in the training subsystem. 

3. Decide whether the risks are so high that no viable set of implementation 
projects could be described to reduce them to an acceptable level. If 
this is the case, then rejection should be considered at this point. If 
potentially high risks exist, then thought should be given at this time to 
nsk reduction projects.. to.-,t\e detailed in Task 4. If the projected project 
cost seems unreasonably hi gh>, i n relation to the benefits associated with 
the reduction of risk, thenthis is perhaps another reason for rejection 

at this time. 
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TASK: MAKE PRELIMINARY FEASIBILITY PROFILE 
SUBTASK: ESTIMATE ATTITUDINAL ACCEPTANCE 



PROCEDURE 

1. On the basis of historical precedents or current expressions of attitude 
about issues represented by the innovation, determine the degree to which 
users of the innovation are likely to accept the innovation as desirable, 
meaningfu.1 , and appropriate to their sense of role and of self-image 

.. General experience has shown that if the instructor rejects the instruc-'' 
tional device, function, or role, he will transmit this attitude to most 
of his students. This does not mean that the converse occurs, namely 
that instructor acceptance is tantamount to student acceptance. The 
assessor should recognize that some resistance may be either transient 
or a deep and long-lasting resentment. 

2. Select the statement below which best describes the kind of attitudinal 
acceptance to be expected if the innovation is introduced:. 

( ) The instructional conniunity will almost certainly respond 
to the innovation with sustained hostility. 

( ) The instructional community will probably respond with some 
moderate but temporary resistance to the innovation. 

( ) The innovation can be introduced to the instructional 

community with little or no feeling about it, one wav or 
the other. ' "' . 

( ) On the main, the instructional community will welcome thl" 
innovation, especially after its realistic values have tiieen 
experienced. • 

3. Decide whether the risks are so high that no viable set of implementation 
projects could be described to reduce them to an acceptable level. If 
this IS the case, then rejection should be considered at this point If 
potentially high risks exist, then thought should be given at this time to 
risk reduction projects to be detailed in Task 4. If the projected proiect 

\ cost seems unreasonably high in relation to the benefits associated with 

N the reduction of risk, then this is perhaps* another reason for rejection 

\^ at this time. 

\ 
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TASK: MAKE PRELIMINARY FEASIBILITY PROFILE 

SUBTASK: SUMMARIZE PRELIMINARY FEASIBILITY PROFILE DATA 



PROCEDURE 

1. Assemble the data from Task 3, Subtasks 1 through 7 into the format 
suggested in Figure 3.8.1. Check the level of risk and importance-without 

' considering additional risk reduction projects. 

2. Indicate the number of risk reduction projects required and identified 
for each risk category. 

3. Estimate the approximate total cost for risk reduction projects in each ■ 
category. f 

4. Sunmarize the risk data with an overall risk assessment, total for risk 
reduction projects, and estimated total cost for the projects. 

5. An extremely useful exercise at this point would be to reevaluate the 
individual risk levels considering successful completion of all the 
identified risk reduction projects. 

COMMENT , . • 

The data summarized here may prompt a rejection decision at this point, how- 
ever, whether a rejection occurs or analysis continues, the data will be a 
valuable input to the management decision process. 
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04 00 02 

TASK: PERFORM ANALYTIC FEASIBILITY ASSESSMENT , ' 

Having decided as the result of Task 3 that there are no insurmountable 
obstacles in achieving the benefits of the proposed innovation, more rigorous 
application of the factors outlined in Task 3 will be made at this stage of 
the assessment process. This task has a two-fold purpose. First is to refine 
the analysis of Task 3 in developing success probabilities for inclusion in 
the decision -tree. Second is to further define implementation projects ' 
targeted at reducing the risk from any of the factors considered in Task/3. 
Once risk reduction programs have been defined, the success probabilities are 
reestimated and used along with the original probability estimates to cdmplete 
tne decision tree. If no major problem areas were identified, then Task 4 
would be bypassed; although the factors considered in the analysis of ytask 3 
shou d be reassessed after Task 5 (Determine Range-of-Effect) has beeti 
completed, since a broadened application of the innovation may expose new 



SUBTASK: 



REFERENCE 



Identify the factors from Task 3 that /represent potentially 04.01 01 
high risks in the implementation and use of the proposed 
innovation. (This and the following subtasks are also 
applied to each alternative resulting from the analysis in 
Task 2.) ' 

2. Evaluate the probabilities that the innovation can be 04 02 01 
successfully implemented and used considering the identi- 
fied risks. I 

] 

3. Define projects and associated costk intended to reduce / 04 03 01 
risks to a reasonable level. \ . u . . , 

4. Reevaluate the probabilities of sucbessful implementation/ 04 04 01 
and use considering achievement of jiiroject results. 



Make preliminary estimate of potential scope of applica- 04 05 01 
tion (range-of-effect) required for j^roject payoff. 

Create tickler file entry identifying\criteria for ' 04 06 01 

reassessment of innovation if, at this\ point, the pro- ■ ' 
posal does not appear feasible. \ 



\ 

\ 
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TASK: PERFORMN^ALYTIC FEASIBILITY ASSESSMENT 
SUBTASK: IDENTIFY POTENTIALLY HIGH RISK AREAS 



PROCEDURE 

r. Refer to the output of Task 3 which identifies risks which may impact the 
projected implementation or use of the proposed innovation in the follow- 
ing topical categories. 

t Importance of innovation to projected Navy mission 

t Organizational compatibility 

t Goals/policy compatibility 

t State-of-art (Technological feasibility) 

•t Funding constraints 

t Technical support capability 

t Atti tudina^^l acceptance. 

2. Develop further detail for each risk statement which describes the specifi 
obstacle to be overcome (the potential problem envisioned in implementing 
and/or using the innovation). The detail should suggest R&D studies or 
projects which can be undertaken to reduce the overall risk in each major 
category. More than one approach to reducing the risks may be identified 
and each reasonable alternative should be described and carried through th 
subsequent analysis. 

J. Document results of analysis and incorporate into project file. 



Er|c . . III-51 



\ 

\ 



TAEG REPORT NO. 12-3 



TASK SUBTASK PAGE 
04 02 01 



TASK: PERFORM ANALYTIC FEASIBILITY ASSESSMENT 
SUBTASK: EVALUATE SUCCESS PROBABILITIES 



DESCRIPTION ^ 

Evaluation of success probability is performed with respect to implementation 
of the Innovation as well as in relation to its acceptance by the using system. 
Some of the risk categories from the preceding subtask will affect implementa- 
tion; e.g., state-of-art, others will affect innovation use; e.g., funding 
constraints. In this procedure the risk items are separated so that an inde- 
pendent evaluation can be made for each category. The process outlined in this 
procedure may require several iterations using judgmental Inputs from a number 
of experts in the specific problem areas. Also, since the impacts may not be 
independent; e.g., the resolution of. an organizational incompatibility may 
change the probability for a technical support type problem; the interrela- 
tionships and conditional probabilities may have to be worked out using extended 
decision tree diagrams or some form of cross-impact tables*. The basic decision 
tree format discussed in Task 3 and shwn in Figure 4.2.1 will be used as a start- 
ing point. 

PROCEDURE 

1. Classify the risk statements from 04.01 into: 

a. Those that will create problems in the implementation of the Innovation. 

b. Those that will interfere with the successful use of the innovation 
once it has been implemented. 

2. Given no specific project action is undertaken to resolve any of the prob- 
lem areas (other than a continuing level of awareness that the problem 
exists), estimate the probability that: 

' a. Successful implementation (includes any development and installation 
activities) can be achieved (U in Figure 4.2.1). 

b. The user will accept the application of the innovation as planned 

(V- in Figure 4.2.1). User in this context refers' to the using system; 
i.e., organization, function, environment, etc. 

COMMENT: Composite probabilities where the problem areas are -not independent 
may require more complex analysis than is suggested here. 



*Paper on DELPHI Techniques and Cross Impact Analysis - Author unknown. 
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3. For each oF the above two est'imates, provide an interval estimate where 
the one end of the range is the probability of success under the most 
favorable conditions and the other end of the range is the probability of 
success under the most adverse conditions. 

4. Document results of these estimates for incorporation into the project 
f i 1 e . 

RATIONALE 

The probability estimates and confidence intervals developed in this subtask 
will be part of the data used to determine the relative value of the decision 
to either: 

a. Reject the proposed innovation outright. 

b. Accept it without additional study. 

c. Accept it and conduct continued studies considering their associated 
costs. 

The confidence intervals will be used in conjunction with sensitivity analyses 
to determine if additional information should be acquired in order to improve 
the confidence in the estimate. 
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TASK: PERFORM ANALYTIC FEASIBILITY ASSESSMENT 
SURJASK: DEFINE IMPLEMENTATION PROJECTS AND COSTS 



DESCRIPTION , 

The assessor must decide whether the level of risk identified is potentially 
too high to be acceptable. This general reference resulted in. the previously 
identified risk statements. The task, at this point, is to define specific 
projects targeted at overcoming some of the problems which create the poten- 
tially high Jevel of risk, and to cost out these' projects for inclusion in 
the decision tree outcomes resulting from the decision to carry out the pro- 
jects. The preliminary cost estimates performed in Task 3 will be refined 



1. Prepare a project description for each R&D effort, study, experiment, etc., 
which should be considered in reducing a specific problem or group of 
problems to a reasonable level of risk. 

2. Identify resource requirements and approximate time schedule for accom- 
plishing each project. 

3. Prepare a detailed cost analysis of each project. A sample form for 
documenting cost data is shown in Figure 4.3.1. 

4. Group projects into one or more packages, each of which is intended to 
reduce the overall* risk to a reasonable level. 'Where alternative approaches 
to reducing risk have been defined, groupings of projects can be structured 
to include the various alternatives. Different cost/benefit patterns will 
undoubtedly .result from the alternative risk reduction packages and the 
effect upon the decis'lon variable will be determined when the decision tree 
is folded back as outlined in Subtask 04.05. 

5. -Document results of project defini^tion and costing for incorporation into 

the project file. 



at this point. 
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FIGURE 4.3.1. .COST/SAVINGS DATA SHEET 
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TASK: PERFORM ANALYTIC FEASIBILITY ASSESSMENT 
SUBTASK: REEVALUATE SUCCESS PROBABILITIES 



DESCRIPTION 

This involves procedures similar to those outlined in Subtask 04.02. 
PROCEDURE 

1. Given that the specified risk reduction projects are undertaken for each 
grouping or package of risk reduction projects, reestimate the probability 
that the projects will succeed. The values resulting from the estimates 
'Will be assigned to the "projects succeed" leg on the decision tree T for 
project package 1 in Figure 4.2.1). The value for "projects fail" (l-I) 
IS also entered on the decision tree. If more than one project package 
has been defined, similar entries will be made for each additional package. 

2. Given that the projects succeed, for each project package estimate the 
probability that: 

a. Successful implementation can be achieved (W in Figure 4.2.1). 

b. The user will accept the application of the innovation as planned 
(X in Figure 4.2.1). 

3. Similarly, given that the projects fail, for each project package estimate 
the probability that: 

a. Successful implementation can be achieved (Y in Figure 4.2.1). 

b. The user will accept the application of the innovation as planned 
(Z in Figure 4.2.1). 

C0^?*1ENT: In this reassessment of risk, consideration shall be given to the 
time required to complete identified projects from the standpoint that a 
change in implementation date could in itself modify the probabilities of 
success. Where a number of interrelated projects are being planned, some 
leve.l of project scheduling may be required using PERT, CPM, etc., before ■ 
reasonable estimates may be made. 

4. For each of the above estimates, provide an expression of confidence by 
estimating in each case an optimistic value given the most favorable 
conditions occur, and a pessimistic value considering the most adverse 
conditions come about. 

5. Document results of these estimates for incorporation into the project 
file. 
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TASK: PERFORM ANALYTIC FEASIBILITY ASSESSMENT 
SUBTASK: . MAKE PRELIMINARY '^RANGE-OF-EFFECT" ESTIMATE 



DESCRIPTION 

If the process of describing the innovation and conducting the preliminary 
feasibility analysis has not provided a clear basis for a decision regarding 
the innovation'/ this subtask will provide an initial quantitative basis for 
the selection of a decision route. 

PROCEDURE 

1. Using the target applications for the innovation from Task 1; i.e., course, 
jobs, equipments, students, etc., make a preliminary estiitiate of the ex- 
tended range of courses, jobs- etc., over which the proposed innovation 
might apply. 

COMMENT: If data bases are available for searching, preliminary search para- 
meters may be defined without the full analysis required by Task 5 (Determine 
Range-of-Effect) . 

2. Identify the appropriate cost categories impacted by the innovation. A 
further subdivision of costs between development/implementation and re- 
curring operational costs should be maintained so that costs can be re- 
lated to the specific outcomes of the decision tree. For example, one 
class of costs may be expended in the process of attempting to implement, 
and even though the implementation was unsuccessful, these costs will be 
incurred and identified in outcome C of the decision tree shown in Figure 
4.2.1. Shutdown costs in case of failure should be considered. 

3. Prepare gross cost/savings estimates for each cost category considering 
each outcome of the decision tree in Figure 4.2.1. 

COMMENT: If a cost model is available, input parameters should be specified 
arid the model run to determine preliminary costs/savings. 

4. Enter costs on the decision tree for outcomes A through D and specify the 
probabilities (most likely) at points U through Z. An example of a 
completed decision tree is shown in Figure 4.5.1. 

5. Fold back the decision tree to obtain the preliminary value of each of 
the three decision paths. The steps in the foldback process are illus- 
trated in Figure 4.5.2. 
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6'. Select preliminary decision path. The highest positive value on legs 1, 
2, etc., would be an indication of the best preliminary decision. If all 
of these values are zero, based on this preliminary assessment the proposed 
innovation should be rejected. It 1s possible, however, that the identifica 
tion of alternative risk reduction projects which produce revised success 
probabilities could change this preliminary result. Also,' a revised range- 
of-effect analysis may expand savings beyond cost to create a more favorable 
decision outcome. 

7. Document results of decision tree analysis for incorporation into the 
project file. 
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TASK: PERFORM ANALYTIC FEASIBILITY ASSESSMENT 
SUBTASK: CREATE TICKLER FILE .ENTRY 



PROCEDURE ^ - 

1. If a decision is made at this time not to proceed^ the problem areas should 

^ be reanalyzed to establish the basis upon which the innovation might be 

reactivated. For example: ' . ■ " 

• When might presently insufficient funding be available. 

• When will an appropriate level of technical support be available. 

• When might existing organi2!ational - constraints be modified. 

§ When mIgKt attitudinal acceptance on the part of the user population 
be improved to bring about a more acceptable level of risk. 

• When might, a particular technological breakthrough occur. 

2: Create a tickler file entry which reflects an estimated date where risk , 
from identified problem areas may be reduced so that thfe proposed :jnnova- 
tion may now be acceptable. , ' ' 

RATIONALE ' ^ 

One of the objectives of the ETAM procedure -is that information not be lost 
either because of summarization exercises or due to a gap in evaluation or 
implementation of the proposed innovation. rChanging conditions may create 
conditions where application of the innovation appears m.ore reasonable. *It 
is important that a system of estimating the time at which these conditions 
may occur, and a system of forcing visibility of the proposal at that time, 
be instituted. This procedure, while seemingly trivial, is intended to pihe- 
vent loss of perhaps valuable data. ' ' ^ 
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PERFORM COST-BENEFIT ANALYSIS 
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MANAGEMENT 
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5.18 



ASSESS THE 
TRAINING 
BENEFIT ^ 
PATTERN 



THE JOBS BEING 
ASSESSED ARE WITHIN 
TRAINING D&A FUNC- 
TIONS RATHER THAN 
OTHER FLEET/SHORE 
JOBS, 
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TASK: DETERMINE RANGE -OF- EFFECT 

This is likely to be ^he most important and difficult task in the entire- 
procedure. Its general purpose is to determine the full range of potential 
applicability of an i]nnovation in training, operations, and in the personnel 

subsystem. These effects may consist of job performance and training ob-- 

jectives for which "benefits" are yielded, The effects fflay also be potential 
savings of displaceable dol Tars , "time , level of talent required for instruc- 
tionaT function, or/other cost element. The latter will be called "costs and 
cost analysis . Where possible a benefit in achieving a training objective 
(a knowledge or skill) will b'e translated into training cost savings. 

The scope of range-of-effect consists of many factors, concepts and operations.' 
Simplifying maps may be misleading if taken literally or as complete in them- 
selves. Nevertheless, the reader is given several levels of abstraction for 
Task 5: 

'The schematic diagram of major procedural functions that is presented 
at the head of this chapter. 

The listing of subtask names which immediately follows these introductory 
paragraphs. 

The listing of subtasks and variables and categories within each subtask- 
this list follows the subtask name listing. 

The explanatory descriptions of' subtasks and categories within subtasks 
that make up the body of Task 5. 

An Appendix a which treats in analytic detail some of the key characteriza- 
tions in the main text of Task 5. ) 

^^f/^^J^r is not advised to use the contents of the first three items above as 
self-sufficient either for critiquing or for using, the Range-of-Effect procedures, 

The quantitative objectives in determining fange-of-effect in the results of 
training can be expressed in student training hours to achieve a training ob- 
jective, minimum levels of student aptitude requTFelTients . attrit ion rate, and 
level of student performance . These variables are not independent of each 
other, so assumptions mustlje made as to which will be fixed, and which will 

rpipw.^nt . rh'''^''"^?- P^'^P"^"- By determining the range of jobs and tasks 
relevant to the innovation, the range of courses and numbers of students can 
De Identified. 

A second class of quantitative objective is the determination of displaceable 
costs which the innovation might achieve in the Navy's inventory of instruc- 
tional vehicles, including instructors, or to displaceable costs in training 
development and management functions. i-rdining 
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The range-of-effect procedures consist of two major parts. One part consists 
of classificatory terms from which the assessor can select descriptors which 
fit the innovation, and which can be used to search the data base of jobs and 

job tasks, courses and course objectives, instructional vehicles, and their ; - 

properties The results of the. .data-base searches, when checked by the assessor, 
reprepnL the f 1^11 patterr^^o^^ students, courses, instructional operations, 
vehicles of delivery, and other elements to which the benefits and cost advan- 
tages, if any, of the innovation could appropriately apply. 

Note that the "data base" may exist. in the head of the assessor and his asso- 
ciates, on paper documents, or in computerized files for selective access. 

Subject matter search terms are never completely exhaustive or unambiguous 
Human judgment finally determines whether the search operation has indeed made 
a useful match between the two entiti'es described; the properties of the 
innovation and the properties of the entity to be matched with it. Even though 
two descriptions may match each other, the objects denoted by the search may, 
when examined in their fuller contexts, have important differences that over- 
ride their similarities. The user should be aware of the limitations of words 
in describing things. 

After identifying the list of entities to which the innovation has potential 
relevance (courses, job tasks, aptitudes or instructional vehicles, or train- 
ing development and administrative functions), the second major activity for 
the assessor is to assign magnitudes of benefit likely to be conferred by the 
innovation. He also indicates the likelihood of achieving the benefit by giv- 
ing an expression of confidence to. his magnitude estimate. If the innovation 
offers several kinds of benefit, he rates the relative importance of each 
The assessor also identifies and evaluates secondary benefits to be expected 
of the innovation, and liabilities which its adoption might bring about. 

The formats on which the assessor .develops all of the information in this task 
are intended to serve as procedural aids to him. They also transmit the data 
he generates into the cost analysis and business assessment operations that 
lead to an assessment outcome. , 

Because the assessor must select his pathway through range-of-effect analysis • 
by knowing what is ahead, of given choice points of route, he must have a good 
working knowledge of the entire classificatory scheme in this task. Arriving 
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12. 

13. 

14. 

15. 

16. 

17. 

18. 

19. 
20. 
21, 
22. 
23. 
24. 



-Select vehicle type that f its the i nnovation 
Select class of training objective 
Select vehicle properties 

Complete the creating of specifications for searching 
the= instructional vehicle inventory 

Identification of relevant inventory of training vehicles 
Identify benefit pattern for instructional vehicles 
Develop background for task content training analysis 
Identify the learning objective 
Make preliminary evaluation of benefit importance 

Determine if the benefit applies to skill or knowledge 
-training > ^ 

Select task structure element or task "function" 
Identify relevant jobs and tasks ' 
Select relevant stage of learning 
Identify relevant courses 

Identify relevant training units within courses 
Assess the training benefit pattern 
Select type of knowledge 

Select applicable type of reference knowledge " 

Identify ..type of enabling knowledge 

Identify type of task format 

Assess on-the-job benefits 

Identify relevant function in development and 
administrative management of training 



SUBTASK PAGE 
00 08 " 

REFERENCE 

05.03.01 
05.04.01 
05.05.01 
05.06.01 

05.07.01 
05.08.01 
05.09.01 
05.10.01 
05.11 .01 
05.12.01 

05.13.01 ' 

05.14.01, 

05.15.01 

05.16.01 

05.f>.01 

05.18.01 
05.19.01 
05.20.01 
05.21.01 
05, 22.01. 
05.23.01 
05.24.01 
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TOPICAL OUTLINE OF RANGE-OF-EFFECT , 

This listing should serve as a topical guide to the content of Task 5, and 
after knowledge of the item definitions and descriptions in the main text 
which follows, as a kind of checklist. The subtasks are sequenced by logical 
order rather than by procedural order so as to avoid redundancies. 

05,01 Choose the major route for determining range of. effect 

Instructional vehicles . 

Task content training . . . . 

Development and administrative management of training 

INSTRUCTIONAL VEHICLES 

05.03 Vehicle types / ^ ' 

Instructor 
Static graphics 
Animated graphics 
, Static physical models 
Procedural trainers 

Symbolic ' ^ « 

Physical mockup 

Task simulators - 

05.04 Class of training objective 
Reference knowledge 

Knowledge task specific and enabling 

Task-ski.ll formats ^ • . 

Skill training ' 

05.05 Vehicle properties 
Instructional functions; "1 

Type of stimulus presented to student 

visual ' ! 

auditory. 

kinesthetic 

tactile 
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Type of content displayed * 

text-verbal 

diagrammatic 

representational 

abstracted pictorial representation 
pictorial representations 
^ . physical representations 

Type of presentational sequence 

library of frames or items 

presentation sequence not applicable . 

fixed sequential frames or Items 
^ random selection of frame sequences 
V dynamic change' of coptent within frame 

Selection source for sequencing 

internal program 
instructor 
student choice 
student perform^ince 
cpmbi nations of the above 

Type of external control operated by student 

not applicable directly 

artificial or symbolic response by student 

representational response by student 

by symbolic selection 

by dummy control activation ' 

task-manipulative response 

non-dynamic in time and force 
dynamic and interactive. 

Feedback presentation logic 

not applicable 

selects next stimulus item or sequence 
gives evaluation of preceding response 
selects and presents guidance information 
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Response evaluation logic 



not internal — depends on Instructor evaluation 
evaluation limited to student's inmediate response 
evaluation extended to a set of student^ responses 
tolerance limits on acceptable student response 

fixed 

.variable ^ 

Utilization f-unctions ; 

t? • , • 

Programmability, and control of device operation 
Portability, including storage 
Multiple usage 
, . ^ \ Set upland operating simplification 

-^'^ Maintenance simplification; dependability. 

Updatability in instructional function 
Comtxdning instructional functions 
Modularization of training' content ^ 

Direct- translatability from a/requirement to training decisions 
- and programming of training content 

Supplemental training functions 

Extended range of control of learning^conditions 
Extended range of subject matter for practice 
Extended range of task functions trained 
Instructor roles facilitated . 

.TASK CONTENT TRAINING 

05 . 1 3 Task structure element or task function 
Goal projection 

Scan-detect ^ 
Identify 

' \ Interpret ' ^ 

Procedure following ' . 
- . Decide 

Construct-plan 
Track 

Motor performance 
Interpersonal interaction 
Recall task-cycle information 
RecalT enabling information 
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05.14 Relevant Jobs and Tasks • 
^',Gross job categories 
Operations 

Maintenance ^ ' 

Service 

C&nmand • ^ • ^ 

'Objective task vari^ables 

Equipment and objects used . , , 

Environments in which is taskT performed 

Tools used * ^ " , 

Reference information used in performing task r 

Task operations C'*task structure elements") 

Criteria of task performance 

05..15 Stage of Learning 

Orientation and familiarization 

Task nomenclature (identifications .^locations and names, facts, 
rules) 

Task formats at the 'conceptual levej 

Procedures learned ^at the verbal level. onl'y 

Performing task components with guidance / 

Doing the entire task/job prac,edurally: barely acceptable mastery 
Highly proficient task/iob performance in- work context 
Unusual task conditions " ' » 

Performing job/tasks at key man level 
Refresher learning 

05.19 Type of toowledge 

Reference 
Enabling 

05:20 Reference knowledge « • 

^System' purposes 
Organizational roles 
Contexts of operation 
J , Organizational rules and constraints 
* Npn-work directed content - 



05.21 Enabling /knowledge 



Operational goal' criteria 

Nomenclature, identification and location' of work objects 
Prpcedure descrfpttons 
Job relevant facts , - 
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05.22 Type of task format 

Procedure formats , - ^ 

Decision formats 

Construction formats . • -J 

05.1vS Benefit pattern in training: variables 

Training time reduced ' 
Aptitude level reduced 

Attrition level reduced . ^ ' ■ ^ 

Performance level increased, extended J 

05.23 Benefit pattern on-the-job 

. * Greater flexibility in assignment 

Less on-the-job training to acceptable performance 
Higher productivity le,vels, and better quality, fewer errors 
Greater range in work competence 
Willingness,. under stre[ss 

Higher level of s,el f-fni tiated and self-directed activity 

i 

DEVELOPMENT AND ADMINISTRATIVE MANAGEl^ENT OF TRAINING 

*05,24 Development and Administrative Management of Training 

Capture and description of ski^Tl ahd knowledge requirements 
.Matching fhe skill reiquiremqnt witK descriptions of availabli 
^ * resources in skills, ^aptitudes to I'earn 



resources 1ri us , ^dpti Luaes lo learn 

Matching the skill requirements wjth available trainingsVesources 
developing training requirements from skill requirements 
descriptions ' 



from skill 

aescripnons ' 
Selecting the training environments 
Slow vs. f^st-paced development i.of training material 
Selection or devising of training modules 
Selection of instructional vehicles and media 



Designing the content of instruction 
Devising Selection procedures forlstudents 
Assessment of the student . \ 

Assignment of -graduates to j^ob poslitions 
Evaluation on the lob • 1 



w,. on the job 

Career development j i 
.Administrative' planning an(^ control 
.^^valuation of lear^.incj'or entropy af 



4 control , 
entropy af the graining sys^t em - 
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TASK: DETERMINE RANGE-OF-EFFECT 

/ • 



SUBTASf:^ CHOOSE THE MAJOR ROUT^ FQR DETERMINING RAN6E-0F-EFFECT 

• Ther;e are three major routes to be followed for the assessment of the innovation. 
Foyowing one route does not exclude the possibil ity of using elements In another 
roiite. The alternatives are: ^ 

Instructional Vehicles. The basis for choosi);)g this aTternative is that 
the innovation aims primarily at reducing a cost in the development or use 
of an instructional vehicle while maintaining a level of function equivalent 
to the vehicle (or vehicle property) displaced by the, innovation. The 
choice of this route jl ©ads to Subtask' 05;02, Develop Background for Instruc- 
tional Vehicle Analysis. ^ • ^ ^ ' 

•. Task Content, Training^; This analytic route, i^ chosen* if the" innovation is 
directed primarily at^^ncreasing the^ effectiveness or efficiency of in- 
structton-akprocesses that may result in; shorter training time to achieve 
a type "of training objective, lower ciptitud'e requirement, lower attrition 
rates, or a higher level of learned performance. The choice of. this route 
leads to Subtask 05.09, Develop Background far Task Content Trai nlng^AnalysIs 

• Development and Administrative Management of Training.' These are funcl(ions 
ttiat deal with the information interfaces to and froni training, such a$- 
training requirements input graduate competence descriptions as output. 
Others deal with such factors as space; selection of instructional media, 
development of training course content, and administrative control. At the 
present stage of formatting these processes, an innovation dealing with any 
one or more functions rfiust be restricted to showing one or mofe kinds of 
cost advantage compared with the reference 'conditions against which the ' 
innovation is to be evaluated. This choice leads to- SubtaTk' 05.24, Identify 
Relevant Function in Development and Administrative Management of Training.; 

PROCEDURE ' ' - . ^ 

1. On the basis of applying the description of the Innovation prepared in 
Task 1 to the considerations above, select at least tentatively, the 
route for assessment of potential benefits from the alternatives: ^ 

Instructional Vehicles 

Task Content Training. - . - - 

Development and Administrative Management of .Training. 

2. '*If the information provided iir-Subtask 05.01 is insufficient for making this 

choice, the assessor should study the. descriptive and analytic content in. 
all subtasks in Task' 5, and then make his choice of route for -analysis . ' * 
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TASK: DETERMINE RANGE-OF-EFFECT 

SUBTASK: DEVELOP BACKGROUND FOR INSTRUCTIONAL VEHICLE ANALYSIS 



DESCRIPTION 

Instructional vehicles range from a printed page of instructions through human 
instructors and to the actual work objects and tools combined with the 
learner and the work environment. This range defies any simple classification 
of instructional vehicle types, purposes, functions, or characteristics. 

Determining range-of-ef feet of an innovation through the route of instructional 
vehicle characteristics should be based primarily on the following condition: 
the innovation is intended to reduce or displace existing or expected costs on 
an existing or planned inventory of instructional vehicles. The cost reduction 
assumes no loss in training benefit, and is not to be evaluated primarily on an 
increase in the training benefit. Thus, the substitution of digital computers 
for analog computers in complex simulators is likely to be based primarily on 
cost comparisons and cost generalizations rather than on training outcome 
-benefits. When the innovation is directed primarily to an increased training 
benefit, the assessor should generally choose the Task Content Training RoU*e 
(05.09) for determining the innovation's range-of -effect. 

The purpose to be served by a classificatory system for instructional vehicles 
is to provide a set of descriptorjs for linking the innovation to the appropriate 
subset of training vehicles in the entire Navy inventory of training vehicles. 
Tiius, if a grossly cheaper way of programming dynamic three-dimensional repre- 
sentations on two-dimensional displays is the proposed innovation, one wants to 
identify all items in tlie invenfory that have visual displays that present or 
simulate three-dimensional moti.on. Potentially relevant vehicles may not be 
restricted to complex training simulator's; they could also apply to knowledge 
trainers that instruct in principles of mechanical motion, or to perceptual 
trainers for the scanning, detection, and identification of terrain information. 

The assessor has two major routes for identifying the relevant training vehicle 
inventory. He should look ahead beyond the choice point and see if any descrip- 
tors in either of the routes make a good first order fit. If so, that is the 
direction he should take, at least at the start. 

PROCEDURE ^ 

1. Become familiar with the conditions for. selecting the instructional 
vehicles route for determining the range-of-effect for the innovation. 

2. Read the description of the innovation given in Task 1. 

3. Examine the terms in the listings under Subtasks 05.03, Select Vehicle 
Type, and 05.05, Select Vehicle Properties, Some one or combination of 
these terms may suggest a direct way of describing the innovation for 
searching the instructional vehicle inventory. 
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4. Do not be 1 , mi ted to the description of the vehicle by any terminology, 
however. Ir there aV^e other terms which can more quickly and compre- 
hensively identify the inventory items, use them. 

^5. If the innovation is, without doubt , limited to an already identified 
piece of equipment, or a medium used in a given way, it is unnecessary 
to go through an elaborate search operatian. Go directly (to Subtask 
05.05, Select Vehicle Properties. It is possible, however, that the 
innovation, either as a device or as a construction principle, may be ' 
applicable to a class of vehicles with which the assessor is not familiar. 
The data base search on selected descriptors would then be a safe bet. 
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TASK: DETERMINE RANGE-OF-EFFECT 



SUBTASK: SELECT VEKlCLE TYPE THAT FITS THE INNOVATION 



DESCRIPTION: 

A vehicle is anything that performs instructional functions; for purposes of the 
present analysis, this includes the instructor. The innovation may apply more 
of less equally to all members of a given type of instructional vehicle; for 
example, interpersonal skills in instructors, or a device for creating, storing, 
and indexing microfiche and other iTnages. The fastest means of locating the 
relevant inventory is to use the broadest name that covers the range, subsetting 
with*>ualifying terms if the preliminary results of the data base search yield ' 
too many irrelevant hits. 

The names in the list of vehicle types cited below should be understood in terms 
of the definitions, cited in the text under PROCEDURE. Eac^i of the terms cited 
may have somewhat different meanings outside of the present context. Many of 
the distinctions made here among the types are primarily derived from the set of 
Instructional Functions" listed under Subtask 05.05, Select Vehicle Properties. 

The major vehicle types consist of: 

Instructor 
Static graphics 
Animated graphics 

Static physical models ' 
Procedural trainers 

Symbolic 
Physical mockup 

Task simulators. 

PROCEDURE 

1. If the innovation has to do with all members, or a large subset of members, 
Of any one of the following types of Instructional vehicles, select the 
term to which the innovation applies. 

The Instructor. Innovations may be directed towards improvements in the 
selection, training, assignment, motivation, and turnover of instructors 
The_"improvement"^should be assessable in terms of reductions of the norms 
against which the innovation is targeted. Less training time to achieve 
a given instructional competence is directly measurable. An innovation 
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that-enabjes, with the normal training ttme and cost, an instructor to 
teach a greater variety of courses must be more indirectly assessed in 
terms of probable dollar value of the capability in the expected train- 
ing environment. If instructor versatility is expected to reduce in- 
structor turnover rates, the expected reduction of rate (and its 
probability) will have to balance against the tangible values in re- 
ducing that rate. 

The assessmfent picture may Include est^ifliates of intangible factors such 
as "instructor satisfaction" or "instructor interest in what he is 
teaching", but should be identified as intangibles not because that 
makes them unimportant, but to identify their intangibility. 

The instructor is, or should be, the key instructional vehicle in t^e 
entire training establishment. He, therefore, justifies large efforts 
to make him effective in his varied teacher roles. .But he may also be 
the target of many well-meaning, poorly worked-through and inadequately 
tested innovations which, had they been assessed on tough-^minded 
criteria, would quickly have been disregarded. It is because the in- 
structor IS vulnerable to many "innovative" influences, most of which 
may be distractors to him, that assessment screenings can be useful to 
him as well as to the establishment. 

The instructor presents information by words, chalkboard diagrams, 
gestures, and sometimes by demonstrating task perfonnance literally or 
symbolically. In the classroom he is effective primarily in teaching 
concepts, principles, strategies, and aspiration goals. He is com- 
paratively inefficient in. dril 1 on material learned by repetitive 
practice (rote), such as facts, rules, nomenclature, perceptual dis- 
criminations, and classifications. When teaching task skills he may 
have limited instructional value as a model to copy or emulate in per- 
forming the task. He may have great value in responding to individual 
differences among students- (if he has this sensitivity) and tailoring 
aspirations and practice content for the student. He may focus or 
expand student attention while engaged in task practice, provide 
supplementary task cues, and supplement normal task feedback with inter- 
pretive feedback relating specifics of what the student did or 
failed to do that resulted in a good or a poor outcome. 

Both his efficiency and effectiveness as an instructor depend on the 
goodness of his implicit process model of the task, his awareness of 
the kinds and stages of learning, and knowledge of the individual 
learning characteristics of the student.^ The instructor can serve as 
a human model in the establishment of positive attitudes. 

Static Graphics. A static graphic is a display on a page or screen of 
a content that does not move. Text, pictures, diagrams, maps, and 
lists fall into this category. Books, charts, foils and overlays, slide 
projectors, and microfiche, are devices in this- category. 



III~80 l-^'l 



TAEG REPORT NO. 12-3 

TASK SUBTASK PAGE 
.05 03 03 



The training objectives for these devices tend to be limited to the 
acquisition of reference knowledge and in some cases to enabling 
knowledge. The latter is true where the target skill is symbolic, 
such as interpreting the symbols on a terrain map, but in these cases 
there must be supplemental instructional functions brought to bear. 

Developmental and production cost for graphic aids is notoriously high 
at ordinary commercial rates. For example, an invention that would 
enable individuals, after a few minutes of training, to create the 
content and control the process for generating an ordinary colored 
slide would have an avera^ge $30 to $40 commercial cost per slide as 
a displaceable target. 

Other cost factors on static graphics include storage and retrieval. 
An instructor may often prefer to make a new graphic rather than spend 
the time and effort required to find an existing one that might have 
served the purpose. 

Animated Graphics.- . These are graphics that present the appeara'nce of 
motion In the content displayed. Television, the motion picture and 
the computer-gellerated display are examples. (There is an important 
sense in which the image being drawn by the instructor on a blackboard 
is an "animated graphic", although its completion makes it a static 
graphic.) 

Animated graphics may transmit concepts of serial dependencies, process 
sequences, and time dependenci.es. As such, they may contribute to 
reference knowledge. They may be task-specific and enabling if they 
provide a mnemonic structure for the later learning of task procedures, 
or for the generalizing of task procedures to a "variety of work situa- 
tions not immediately apparent to the student. In order to be useful, 
the mnemonic structure must parallel the structure of what is to be 
learned as a task skill. 

Animated graphics may be pictorial or diagrammatic. Both are commer- 
cially expensive to create, the diagrajnmatic ones far more so than the 
pictorial. Computers have been progrc^mmed to develop graphic sequences 
on a display tube that may be less expensive than sequences developed 
manually. The Walt Disney studios have an extensive technology for 
creating animation effects. 

• Static Physical Models. A physical model is a three-dimensional rep- 
resentation of objects. Such representations are often called mock- 
ups. Their demonstration effect as a value to training may be 
questioned, although they may offer a kind of reference knowledge. A 
demonstration model of a variety of fighter airplane ceickpits may 
present similarities and differences among them. Three-dimensional 
representations need not be life size. / 
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A physical model may be used to teach a "task-specific enabling know- 
ledge" by providing instruction on nomenclature and locations of work 
objects. 

These devices are generally expensive to build, costly to store, and 
used only briefly in the student's learning scenario'. . 

Procedural Trainers. Procedural trainers not only present the student 
with a display of information, but enable him to make (1) actual or 
(2) simulated task responses with the device with a feedback repre- 
sentation. Procedural trainers may provide 'direct practice in a 
cognitive skill. An example would be training in the cognitive 
aspects of troubleshooting, or in the planning of a military action. 
The procedural trainer may instruct in the symbolic representation of 
a nontracking perceptual-motor task, such as The verbal sequence that 
describes the actions to take in shutting down a p^/ticular type of 
power plant in a vessel. The Matter would be an example of instruc- 
tion in a task-skill format as represented by the student's ability 
to give a verbal description of the procedure, or to simulate the 
steps in the procedurv^ in the physical task environment. 

Procedure trainers may be physical representations of the task en- • 
vironment. The controls operated by the student may simulate actual 
.functions, or act merely as dummy controls, or have a variety of 
feedback cues that range between these extremes. There is some 
growing evidence, and psychological theory, which shows that the 
physical fidelity of control response in the learn:|ng and generaliza- 
tion of most procedures may be very low, at least through the early 
and middle levels of task learning where extensive practice is re- 
quired for "learning what to do next" with a high degree of reliability. 

Where performance data are collected, the large percentage of human 
errors are always assigned to "failure to follow standard operating 
procedure". These findings imply, among. other things, inadequattf^ 
practice by students on procedures as such, and inadequate practice 
'of procedures in full work context if the latter may interfere with 
recall. 

The kinds and amounts of physical fidelity that are necessary and 
sufficient for variou^p classes of procedure training are poorly under- 
stood, and practices used in designing training vehicles for procedure 
training is largely ad hoc. But because the costs of physical fidelity 
are high, and because students should have massive amounts of practice 
in order to acquire reliable procedural skills, the opportunities for 
innovations leading to large cost reductions in overall training are 
perhaps greater in this field than any other except in the determina- 
tion of what is relevant to teach and learn. 

Evaluations of innovations in procedural trainers should take into 
account the Stiages of Learning analysis (Sufatask 05.15), 



III-82 



TAEG REPORT NO.. 12-3 

TASK SUBTASK PAGE 
05 03 05 

Factors applicable to most procedural trainers are identified under 
Selett Vehicle Properties (Sufatask 05.05), which may he used to further 
refxcie the relevant description of the innovation. - 

Simulators of Operational Devices. These devices have "realistic" 
physical or psychological fidelity to the work task situation. The 
controls used by the operator, and contrbl feedback and other features, 
are mten-Jed to make the student respond as if he were performing in 
the real-life situation. The device may also contain a variety of 
sensors and interpreters of student response for evaluation and control 
ot exercise content presented to 'the student. 

Because simulators are expensive devices to develop, program and 
operate, the re^ are many opportunities for d-isplaceable dollars The 
relationship of kind, and degree of fidelity of physical simulation to 
transfer of training is a key cost factor. Fidelity of stimulus input 
is an especially sensitive cost which is understood abstractly, but in 
the specific case must generally be solved by ad hoc inquiry and ex- 
pensive experimentation. 

to 

Unfortunately, the high cost of simulators tends to restrict the nuiSjer 
of practice hours it is feasible to give individual students. Some 
attention has been directed towards the use of less costly "part task 
trainers" to achieve part skills and lower learning levels of skills 
A major innovation would consist of a systematic set of practical guide- 
lines that would specify criteria of minimum physical simulation require- 
I!ci. °f tasks, stages of learning, and degree of 

task integration. Jhe fomulation of such guidelines would necessarily 
rest on a theory or transfer of training at varying levels of learnino 
in various perceptual, cognitive, and perceptual -motor tasks. 



Simulators are also used to evaluate and check out the competence of 
students who are at or' near operational levels of performance. It is 
unclear whether a simulator that is used for evaluation requires any 
properties that differ from those needed for high level training It 
is assumed that the purpose of the evaluation here is that of qualify- 
ing or disqualifying the student from readiness to perform on opera- 
tional equipment. ^v^^^ 

The assessment of innovations in this class of device should take real- 
c+.,if Recount of the effect of training management, instructors, and 
students on the training effectiveness of the device. It is well known 
that If an instructor dislikes the training device he is required to 
use, the student will also tend to dislike or distrust it with a 
corresponding inadequacy in its potency for learning. This could mean 
that a productive innovation might consist of a means of inducing 
favorable attitudes towards effective training devices that may fail 
to evoke favorable attitudes from teacher and student. 
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All of the vehicle properties listed in Subtask 05.05 apply to simula- 
tors. After identifying that the innovation applies roughJy to sim- 
ulators, use the vehicle properties listings to further refine ttie 
characterization. 

2. If the vehicle type appropriate to the innovation does not appear by name 
in the preceding list, search. Instructional Functions in Subtask 05.05. 

. For example, a trainer to i.dentify and interpret auditory signals would ^ 
not be represented in the above list. Trainers in the vehicle inventory 
with this property would be identified by searching under 05.05, Instruc- 
tional Functions:- type of stimulus: auditory. 

3. Assemble the descriptor names that fit the innovation and retain them for 
use in searching the data base of the instructional vehicle in order to 
detennine those items in the inventory that are appropriate to the innova- 
tion. 

4. If thelnnovation deals with a type of vehicle but is restricted to a class 
of training objective » a two-stage description and search will have to be 
made. The secon-d stage that restricts the innovation to a given set of 
training objectives, wil 1 be treated in Subtask 05.04, Select Class of Train- 
ing Objective. 
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TfSK: DETERMINE RANGE-OF-EFFECT 
♦''^UBTASK: SELECT CLASS OF TRAINING OBJECTIVE 



DESCRIPTION 

It is desirable to detemine the widest potential applicability dif the innova- 
tion to instructional vehicles because the economies of scal'e maA trans form' a 
minor contribution to one of major importance. -But the applicatioti of the 
innovation may herve to be qualified within a .vehicle type by several factors. 
It may be restricted to a given class of training objective, although exten- 
sive to all examples within that .objective. The innova±ion,may also be limited 
to a class of instructional environment; thus slides require a slide projector 
and this may imply a space requirement that limits its range of application. 

The types of training objectives listed below are general, and intended only 
to suggest a training context. This level of descriptors, when^ added to the 
vehicle type descriptor, may be adequate to search the instructional vehicle 
inventory list and yield a Reasonable range af candidates for manual culling. 

PROCEDURE 

1. Select the one or more descriptors which, by exclusion of the other de- 
scriptors in the set, properly restricts the range of applicability of 
...the innovation. 

■ • Reference Knowledge (see definition in Subtask 05.20). 

• Knowledge-Task Specific and Enabling (see definition in Subtask 05.21). 

• Task-Skill Formats (for definition see Subtask 05.22). 
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TASK: ' DETERMINE RANGE-OF-EFFECT 
SUBTASK: SEU|^ VEHICLE PROPERTIES 



DESCRIPTION ' - . 

Assume that the choice has been made to identify the applicable instructional 
vehicles in the vehicle inventcfry by the specification of instructional function 
or. utilization function. Presumably, the innovation, represents a functional . 
capability that is fairly independent of the kind of training content which ^the 
vehicle instructs. 

The following list of Instructional Functions is nat arbitrary. It is a fairly, 
complete and definitive set of functions for control of the teaching and learn- 
ing processes. Some instructional vehicles may in themselves ^contain only 
several of these functions. A graphic aid such as a chart, in itself, is char- 
acteri'zed only by -a visual stimulus and a given type of content such as a "flow 
diagram'^ In this^^case, the remaining functions in the list of instructional • 
functions must be performed manually by teacher and student. 

The second list. Utilization Fujictions, ^cannot claim equivalent comprehensive- 
ness,' and might potentially have to be supplemented." Almost certainly a Utiliza 
tion Function would have to be combined with other descriptors, such as, Instruc- 
tional Function or type'of vehicle (Subtask 05.03), in developing a Osable 
search argument to apply to a vehicle inventory listing. 

PROCEDURE 

1. After examining the description of the innovation, select one or more of 
the instructional functions listed below to which the innovation applies. 
Set down both the name of the function (such as "Type of stimulus' presented 
to the student") and the item or characteristic listed under that function 
which applies to the innovation (such as, for example, "auditory"). If no 
item in this list applies, examine the list of Utilization Functions in 
Ste'p 2. 

INSTRUCTIONAL FUNCTIONS . ^ . . 

^ Type of stimulus presented to studisnt 

Visual — for example, the display of simulated landing strip; or a ^ 
flowchart diagram; or printed text* 

Auditory examples, simulated sonar data; sounds of defect in moving 
mechanical assembly; voice transmission through noisy channel. 

Kinesthetic examples, control force. in aircraft stick and rudder; 
torque applied to* a bolt; the feel of a potentially broken human limb. 
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Tactile —'examples, smoothness of surface of a mechanical bearing- 
conducted temperature tajunctions adjacent to one being soldered: 
texti/re of a type of cloth. • » , 



I'ype' of content "displayed ^ 

Text - verbal - examples;, continuous text; formatted text as in job 
Instructions; text in tabular format. - 

Diagrammatic.- examples: flowcharts? PERT diagrams; histograms; logic 
Charts; decisi£»n tables; schematics. ■ • ' ^ ,. 

« 

Representational • • ' 

Comment: Representation, as applicable to instruction, covers a wide 
range of kind and degree. The topic is often in the context of simula- 
tion fidelity where transfer of training value and costs have complex 
..interaction and tradeoffs. As defined here, a "representational" dis- ■ 
pi ay IS one which has softie degree of isomorphic Tdentitv with the 
physical appearance of real-life task objects and situations. The 
classification here, as elsewhere in this section, is based on whatHhe 
student sees, not on how the material displayed to him is generated, 
me following three items roughly cover the range of representational ' 
formats. 

Abstracted pictorial representation: example - a landing runway 
displayed only as a rectangle in changing perspective and size for 
training in aircraft landings. Secondary cues such as terrain 
buildings or other structures, are not presented, but the shapes . 
of the rectangle correctly simulate the appearance 6f runways under 
different landing conditions. In other training contexts, -the 
silhouette of aircraft 'types or types of ships would be abstT^acted 
pictorial represent-ations. ' >r,ai-i,eu 

Pictorial representations: images of actual appearances 'of objects 
and situations such as projected by motion picture cameras, photo- 
graphs or realistic drawings. - 

Physical representations: examples, the actual work object itself 
or three-dimensional mockups. on. 

Type of presentational sequence » 

Library of "framed" or items - the creation of a library servino as a 
source of instructional material, but where the items as stored in the 
library have no inherent sequentiality. 

Pre§'entation sequence not applicable to vehicle - as for example, the 
physical model of 'a device, or a cutaway model. '^Admpie, tne 
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Fixed sequential frames or items — for example, the s£and\rd Kpdak 
carousel, or slide film, or a continuQus motion picture. ■ \ " •• 
■ • . \ 
.Random selection of frame sequences --'sequence may be conditional 
on^choi-ce of user or program,' or based on student response such as / 
in branching type of computer assisted instruction. 

Dynamic change of content within frame by display attributes or 
variables — as is typical of computer controlled simulators where 
altitude indicat,ion may change independent .of airspeed indication, 
or the elements in a display of a simulated display may/ remain con- 
stant but change with respect to simulated point of regard. 

Selection source for sequencing ' • . ' , 

Internal program — for example, a computer' program, the operation of 
which selects a next frame or item where the selection is based on 
student response to a problem presentation; or' a fixed mechanism for 
advancing each next frame in fixed sequence. 

Instructor -- selects what wil.l next be" presented to the student 
either as a specific item or a sequence of related items. 

Student — selects next item of presentation by voluntary choice. 

Student performance — the outcome of the. student's re"sponse to a 
learning- itgm determines what next iten?,will be presented %o hjm. 



Combinations of the above four items. 

Type of external control operated -by student (relevant 'to task oper.atlons) 

' A , ' . - ■ 

Not applicable directly forekmple, because the instructional func- 
tion of interest is not linked t/, a student's response. 

Artificia-l or symbolic response by student — such as when the "student 
selects from multiple choice options presented verbally. 



By symbolic selection — ;^s by pictorial display of control switches' 
keyed to an answer sheet. 

By duminy control activation 'rj physical switches and other controls 
manipulated by student, but which are not operational. 




•Representational, (semirealistic) response by student: 



Task-manipulative response: 



Nondynamic In time and force-wlth respect to changes that take place 
• in the ^presentational display of the taslfr-.^,. 
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Dynamic and interactive fidelity (to some degree) in time and 
control forces with display change. 



• Feedback presentation 'logic 



Not applicable to the issue being investigated. 



Selects next stimulus item or sequence — what is next presented to 
the student depends upon an evaluation of the student's response. 

GiVv<3S evaluation of preceding response to student the instructional 
vehi^cle preseB^ts the student with information that to some degree 
critiques his response or set of responses. This may range from 
"right" or "wrong" to an explanation of why the student's response 
was right or wrong or inadequate. 

Selects and presents guidance information as well as problem data — 
the guidance data presented is conditional upon the student's previous 
responses. 

• Response evaluation logic 

Not Internal depends on instructor evaluation of student's response, 
or the student's evaluation of his response. 

Evaluation limited to student's immediate response, 

Evaluation extended to a set of student response, and/or to classes 
of student response for example, as in a debriefing of a simulated 
mission where the student's general strategy is evaluated. 

. Tolerance limits on acceptable student response; 

Fixed — in terms of acceptable or non-acceptable response. 

Variable such as dependent on amount of practice or stage of 
learning the task. ' 

More than pne of the foregoing instructional functions" may apply to an 
innovation. Furthermore, the innovation may not necessarily apply to a 
mechanized function, but to human activities, in the instruction/learning 
process. 

The reader is reminded that the concern in the Instructional Vehicles 
section of determining range of effect of the innovation is directed towards 
displaceable costs by substituting the innovation for an existing or pro- 
jected alternative. Where the training function is intended to have a 
benefit in what is learned by the student, the assessment should include 
. the analytic route called "Task Content Training Analysis". It is this 
distinction which rules out such factors as student motivation, student 
aptitudes, learning efficiency from consideration in this Instructional 
Q ^ehicles section. 
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Select one or more applicable Utilization Functions from the foU&wihg list. 
Note that whereas innovations may improve the "convenience" of any of these 
functions in a device, it is generally possible to translate convenience 
into some kind of dollar value as a justification. 

UTILIZATION FUNCTIONS 

• Prograrmiabili ty and control of device operation. 

• Portability, including storage. ,5 

• Multiple usage as of job aid as well as training veh-i.cle. 

e Set up and operating simplification requiring less ability and train- 
ing time for the instructor to use effectively, or that enables the 
student to perform the operations. 

• Maintenance simplification; dependability. 
§ Updatability in instructional function. 

•■ Combining instructional capabilities heretofore distributed among several 
media/ devices. 

• Modularization of training-cmtent-^or more general -.af) p 1-1 c^bi-l-i t^- ^ 

• Direct translatabi 1 i ty from a format for describing job task require- 
ments into training decisions and the selection or programming of train- 
ing content and sequence. (This item is also treated in Subtask 05.24.) 

Note that a Utilization Function in itself cannot be effectively used to 
search a data base of instructional vehicles. It will have to be combined 
with additional classi fi catory terms* such as Vehicle Type and, very likely, 
subsets within Vehicle Type beyond the descriptors in the range-of-effect 
section. The device context of the innovation itself may provide more or 
less self-evident characterizations to be added to the search profile, or 
may point directly to the set of relevant vehicles to an individual with a 
general knowledge of the Navy's inventory of instructional vehicles. (For 
an exhaustive listing of media and instructional vehicles, see the report 
by Training Analysis and Evaluation Group (TAEG), A Technique for Choosing 
Cost-Effective Instructional Media , Appendix published April 1974. • 

Identify training functions that may be supplementary benefits to dis- 
placeable costs. These supplementary benefits will be useful in formulating 
the benefits picture for the innovation beyond the primary benefit of cost 
advantage. It should be understood, however, that a formal assessment of 
the innovation's value for an increased training benefit in generating* 
knowledge and skill in students must go through task content training assess- 
ment as in Subtask .05.09. 
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TRAINING FUNCTIONS TREATED AS SUPPLEMENTAL BENEFITS TO DISPLACEABLE COSTS 

• Extended Range of Control of Learning Conditions. Taking learning to 
a higher level of performance or expertise on a task or task-cluster. 
May be by enabling more practice on the same materials to become 
practicable, or by modifying presentation content and units and feed- 
back, or modifying rate of action demand, or by fostering the integra- 
tion of part- tasks already learned. 

t Extended Range of Subject Matter for Practice. This can enable a skill 
to be generalized to the range of conditions encountered In real life, 
coping with contingencies, and adapting to environmental constraints 
and opportunities. 

• Extended Range of Task Functions on Which Training is Given. For 
example, a device specialized for signal detection and identification 
becomes adaptea to instruction on interpretation of the meaning of 
information patterns, or to coping procedures and strategies. 

• Instructor Roles Facilitated. 

a. /The instructor's capabilities may be extended for the purpose of 
sharpening instruction ■ to the individual or group. 

l>7-~e^erfeal requirements are-reduced so that greater time and* atten- 
tion can be given to the teaching/learning process. 
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TASK: DETERMINE RANGE -OF- EFFECT 



SUBTASK: COMPLETE THE CREATING OF SPECIFICATIONS FOR SEARCHING THE 
INSTRUCTIONAL VEHICLE INVENTORY 



DESCRIPTION . . . 

After following the procedures and using the descriptor listings thus far in 
describing the innovation as it might relate to items in the instructional 
vehicle inventory, it may be useful to look at the results with a view of add- 
ing commonsense to the search operation. For example, if the devices to which 
the innovation applies can be used only in classroom environments (and there 
are no formal descriptors which distinguish this kind of training environment)", 
add the term to the search specification. The assessor should be careful, how- 
ever, not to include a term in the search specification that will be improperly 
restrictive and will risk loss of a large portion of relevant devices along 
with the elimination of a few irrelevant ones. If the innovation is a promising 
one for whatever vehicles it applies to, the assessor is advised to follow the 
safe strategy of throwing out a wide net that may capture a fairly large portion 
of items that, on his manual inspection of the context descriptions of the 
device, he recognizes as irrelevant to the innovation and discards as candidates 

PROCEDURE 

1,. Organize and set down all descriptor terms that are relevant to selecting 
applicable candidates from the inventory of instructional vehicles, while 
rejecting those that are not applicable. The descriptors may be drawn'- 
from, but not exclusively from, those under Vehicle Type, Vehicle Property- 
Instructional Function, Vehicle Property-Utilization Function, and Class 
of Training Objective. 

2. Check the descriptor set used in the data base that describes e^ch entry 
in the listing' of Instructional Vehicles for further clues as to relevant 
descriptors that will link the innovation to appropriate vehicles and 
exclude inappropriate vehicles. '' ^ 
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TASK: determine' RANGE-OF-EFFECT 



SUBTASK: IDENTIFICATION OF RELEVANT INVENTORY OF TRAINING VEHICLES 



PROCEDURE 

1. Run the search specification summarized in Subtask 05,06 against the data 
base of the complete inventory of instructional v^ehicles used by the Navy. 
The search operation will attempt to match the descriptors in the search 
specification with the descriptors of each item in the inventory list. 
Those that match will be presented as a listing to the assessor. Accompany- 
ing the name of oach of these "hits" will be an abstract or summarized text 
description of the. vehicle, plus a structured but detailed description of 
the vehicle. The output will (or should) Include other data' relevant to 
the benefits assessment to be made in the next step: quantity, range of 
usage, courses and course objectives used for, anticipated future demand 
rate, and various categories of cost. 

This output listing and accompanying contextual descriptions of the hits 
made by the search specification is only the first stage in developing a 
list of vehicles relevant to the innovation. 

2. On the basis of a manual inspection of the context descriptions of the 
candidate vehicles in the output listing reject those that are. inappli- 
cable, and accept those that realistically fit the full characterization 
of the innovation, 

3. Retain the listings of the relevant vehicles and the context data associated 
with each of them for Benefits Assessment, Subtask 05.08. 
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TASK: DETERMINE RAN GE-OF-EFFECT 

SUBTASK: IDENTIFY BENEFIT PATTERN FOR INSTRUCTIONAL VEHICLES 
DESCRIPTION 

The assessment of benefits in this path of analysis must be limited to deter- 
mining the cost advantages or disadvantages of the instructional vehicle, or 
its property, to which the innovation is relevant. 

There are two considerations in assessing the benefit pattern. One is the re- 
placement of the instructional vehicle by another, which may be the innovated 
device where cost is the sole variable of interest. The second consideration 
is the potential benefit that may be obtained by using the innovation as im- 
pacting one or more of the Utilization Functions (Subtask 05.05). 

PROCEDURE ^ 

1. If the benefit involves the replacement of one vehicle type by another, 
and there are no benefits or liabilities associated with this replacement, 
then the benefit is restricted to the cost differential. No further 
analysis is required if the replacement does not have hidden costs. 

2. If the innovation affects certain attributes of the vehicle for which 
benefits can be identified, the same process should be followed that is 
used in Subtask 05.18 by using the "Format for describing a supplementary 
benefit or liability." 

3. If the benefit pattern of displaceable costs does not appear to justify 
acceptance of the innovation (which would permit going directly to Task 6) 
then additional benefits should be explored via the Task Content Training 
Route (Subtask ^05. 09) using the information collected in Subtask 05.04. 
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TASK: DETERMINE RANGE-OF-EFFfCT 



SUBTASK: DEVELOP A BACKGROUND FOR TASK CONTENT TRAINING ANALYSIS 



DESCRIPTION 

Assume tKat the innovation points to a more efficient or effective set of opera- 
tions for producing a knowledge or skill which may be a training objective or an 
on-the-job performance capability. If the intended outcome can be related to 
existing training objectives, the benefit may be assessed in terms of a cojt 
Tmpr^ovement. In this case, the evaluation would depend on the courses and student- 
hours affected by the innovation in the context of existing objectives on student 
performance levels- 

If, however, the innovation points to performance benefits that go beyond existing 
training objectives, and no cost improvements can be demonstrated from the innova- 
tion, the analysis must be applied to job tasks ^in the Navy's operational milieu. 
If the innovation increases training costs, the estimated benefits of the added 
performance capabilities must be assessed as worth the added costs or not worth 
. the added costs- 

The following steps for determining, by means of task descriptors, what training 
objectives are relevant, also apply in principle to determining what tasks in 
the operational environment are relevant — assuming no training objective exists 
that is relatable to the operational tasks or jobs in question. 

Assume that the learning outcomes of i:he innovation have been verbally described. . 
The assessing officer must pro*bably be familiar with the learning processes and 
outcomes of the innovation in operational terms, as well as with the innovator's 
descriptions of conditions and outcomes. This substantive knowledge may enable 
restrictions, as well as amplifications, to the verbal range-of-effect perceived 
by the innovator. Thus, an innovation that the innovator perceives as appli- 
cable only to the medical student's learning of the names, positions, ^nd actions 
of muscle and bones in the body, may be generalized to the integration of in- 
struction on nomenclature and identifications of arw sets of work objects, and 
also for instruction in the verbal stages of learning sequential procedures of 
anyH cind. The following sets of categorizations should foster the making of 
ITiTs kind of generalizing insight, as well as to place proper restrictions and 
qualifications on the range of applicability. 



1. Become familiar with the conditions for selecting the task content analysis 
route for determining the range-of-effect for the innovation. . 

2. Study the description of the innovation in Task 1; the statement of task 
objectives, training objectives, and especially the context of the empirical 
findings as reported in Task 1. 



PROCEDURE 
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3. Examine the terms in the various subtasks from 05.12 through 05 21 Some 
one or combination of these terms (and their reference definitions in the 
text describing the subtasks) may suggest a more direct way of describing 
the innovation for searching through the inventory of Navy training courses 
and the inventory af Navy job tasks, than the formal, route structure might • 
require. Notice that each choice of a descriptor along the route both opens 
a corridor of relevant applicability of the innovation, and at the same 
time, closes off other potential corridors as irrelevant to the innovation. 

4. Do not be limited to the description of the tasks and courses by any ter- 
minology appearing' in the subtask analyses. If you can confidently go 
directly to the names of all the courses clearly relevant to the innovation, 
hL!?:,.i '-^^ are otlTer terms which will more quickly, and compre- 
hensively. Identify the jobs and tasks, or courses,^ use them. 

^' ^V^^-^"T^*^"°^■'^' without doubt , Umited to an already Identified job 
and task within the job, or .an^^entifiable course and course 'objective, 
it unnecessary to go through an elaborate search operation. Go directly 
to Subtask 05.18, Assess the Benefit Pattern. It is possible, however, that 
the Innovation, either as instruction or as a result in a skill level, may 
fL?ff \ ^J"^ °^ cours.es and jobs with which the assessor is not 

Tamillar. The data base search on selected descriptors would then be a 
safe hedge against this unfamil iarity . 
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TASK: DETERMINE RANGE-OF-E,FFECT 



SUBTASK: IDENTIFY THE LEARNING OBJECTIVE 



DESCRIPTION 

The "primary source of information for specifying the learning objective(s) , 
preliminary to more formal analysis, is the description of the innovation 
made in Task 1. That desQription may be supplemented and qualified inform- 
ally by training experts who have operational knowledge about the innovation 
and any studies performed with it, or more detailed knowledge about the jobs 
and tasks targeted by the innovation than was known to the innovator. 

PROCEDURE 

Modify or extend the description of the innovation as appropriate about the 
training product or process to which it applies, and the task and task context 
,to which it applies. 

COMMENT 

The procedures in range-of-effect will result in further analysis of the innova 
tion, and some of those results may be redundant to what is done in this sub- 
task. However, sinfie the description of the innovation made here will be a 
major starting point and reference for further analysis, the completeness of 
the description made here, including restrictions and qualification, should 
facilitate later steps. 
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TASK: DETERMINE RANGE-OF-EFFECT 



SUBTASK: MAKE PRELIMINARY EVALUATION OF BENEFIT IMPORTANCE 



DESCRI'PTION 

This subtask has two major parts:' 

a. Is the training objective to which the innovation is limited of minor, 
moderate, or essential importance? If the innovation is directly tied 
to a class of task such as the rigging of models-of sailing ships, it 
may be possible to discard the proposal quickly. "Note whereas an out- 
of-hand rejection of an innovation at this point might save .further 
assessment work, it could lead to overlooking a significant instructional 
principle. In general, instruction in a nonskill directed knowledge 
should hai'e less importance than skill training. 

■ b. Is the magnitude of the found or potential improvement in training benefit, 
or achieving the training objective, small, moderate, or very substantial? 
Wh'6n humans must convert a principle or reference model into actualities 
in their own contexts, a substantial degradation of the potential benefit 
of the principle tends to occur. A benefit of, say 205^ shown in a labora- 
tory, may shrink to just a few percent, or zero, when applied by average 
practitioners. Exceptions may occur. " ---s 



1. Examine carefully the description of the innovation, and the extended 
description of the kinds of training objective towards which the innova- 
tion appears Restricted, as described in Subtask 05.10. Take into account 
the learning requirements of the student in his pattern of learning and 

. fulfillment of training requirements, but consider even more seriously the 
importance of the instructional objective (as characterized thus far). 
Select the answer which best fits your conclusion: 

The training objective to which the innovation is targeted seems, ' 
in terms of the training mission and operational mission of the 
Nivy, of: 

a. Identifiable but minor importance 

b. Moderate importance 



PROCEDURE 



c. 



Very substantial importance 



d. 



Crucial importance. 
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■2. Examine the empirical data or other basis for estimating the magnitude of 
the incremental benefit in training value of the innovation as compared 
-with the existing or reference condition, '.ake fnto account not only 
data averages but their distributions (standard deviations). Select the 
answer which best fits your conclusion: 

The magnitude of the benefit, taking into account all factors that 
could have practical bearing on the results, is: 

a. . Of no practical importance 

b. Of identifiable to moderate importance 

c. Of very substantial importance 

d. . Of almost crucial importance. 

3. Alow or negative score on both of these questions could lead to an inme- 
diate termination of the assessment process with a rejection of the inno- 
vation. However, even though the assessor had well in mind the major 
variables in training and in on-the-job performance requirements, and he 
ran this internalized model against his understanding of the innovation, 
the risk of losing a good bet might be small, but not nonexistent. 

" If the innovation is rejected at this point, the work done thus far should 
be documented and entered in the data base of rejected innovations. 

4. If the innovation passes through this test, proceed into the determination 
of range-of-effect and formal benefits analysis. Proceed to Subtask 05.12, 
Determine If the Benefit Applies to Skill or Knowledge Training. 
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TASK: DETERMINE RANGE~OF-EFFECT 



SUBTASK: DETERMINE IF THE BENEFIT APPLIES TO SKILL OR KNOWLEDGE TRAINING 



13ESCRIPTI0N 

Determining range-of-effect for knowledge training is a different route than 
range-of-effect for skill training. The descriptors along each route should 
be the primary bas-is for the selection of the route. These descriptors are 
intended to provide the basis for the distinction. Skill training is actual 
practice on an operational work task, In actual or simulated conditions. Note 
that different stages" of skill learning may call for different learning condi- 
tions. A skill is a doing or ability to do; it has specific references in 
equipment and other work objects, environments, tools, reference information, 
andcriteria of goodness. Skills may be symbolic and cognitive such as 
decision-making as well as manipulative, procedural as -well as tracking. 
Training that is not skill-oriented is,, by definition, "knowledge" oriented. 

It should be. repeated that making the classi-fications that follow ts not an 
end in itself, but a- means to find a set of descriptors that fit the range of 
training objectives and training .and" learning processes appropriate to the 
innovation's potential utility. The necessary and sufficient training supports, 
for the learning of a tracking skill, for example, differ from those generally 
necessary and sufficient to teach and learn a procedural skill, which in turn 
differ from decision-making and constructing skills. Each category implies' a 
training resource that differs in kind or magnitude for a training resource 
both efficient and effective for another category of skill. This statement 
applies mainly to the difference between skill and knowledge pathways, and to 
terms listed under Task Structure Element, Task Formats, and Stage of Learning. 

Distinctions are made here between "Reference Knowledges" and "Enabling Know- 
ledges." The latter a^ skill-oriented but may be acquired through cognitive 
and symbolic processed, rather than by actual practice on the task itself in 
some real or simulateijl task setting. 



The distinctions 
topics'. 




arized here are clarified in the body of the respective 
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TASK: DETERMINE RANGE-OF-EFFECT 

SUBTASK: SELECT TASK STRUCTURE ELEMENT OR TASK, "FUNCTION" 



DESCRIPTION 



This is another step in selecting descriptors that will 'pin down the applica- 
bility of the innovation,' while at the same time probing for the extent of 
its range. " ■ • . 

A skill, generally,, is made up of a number of components. Although there are 
many ways &f classifying or subdividing components', the set of descriptors 
•used here is related to instructional and learning properties. Each task 
element denotes a class of capability that KaV somewha.t different 'demands on 
what IS necessary in an instructional capability, than is required" for another 
task element. Thus, Q.learly, learning to scan a piece of equipment and detect* 
burned insulation, broken wires^ or-l-eaking receptacles, demand-s a lower mini- 
mum effective instructional capability than would be essential to learning to 
l^^f^^^fy the damaged wire or leaking receptacle, .or interpre t the cause of the 

damage or- its effect on the performanc.e of the devi'-ce^ 

■ gf 

In this subtask, we are trying to narrow the relevaVit appl icabi l.fty of the - 
Innovation. The result of this step will have shar.pened the de'scription of 
the conceptual applicability of the innovation, in. readiness to apply it to a 
search of the job task inventory for reasonable matches to the specification ' . 
.developed thus far. / ' . • 

The name of the relevant /ask structure/el ement, in itself, can sometimes serve 
as the principal search^escription applied to an inventory of training courses 
and training objectives, or to an inventory, of task descriptions. Thus, a train- 
ing technique for the "Identification of figural elements in complex perceptual 
fields may apply to a sk/11 in "identifications" from reconnaissance photos to 
visual checkout of an electronic chip layout. 

The following is a brief list of what may be called task structure elements or 
functions. _Although the set is small, it covers a wide range of real-life tasks' 
and especially if the items are taken in combination. Thus, scan-and-detect 
may be combined with identify, identify with interpret, interpret with decision- 
making, and so on, in order to match with the training or job objective in the 
innovation: = ^ j,. ^ 

The following points should be clearly understood: ■ . • 



Every work activity will be represented to some degree by. all of these task 
functions. But in most tasks as performed in a given situHtTonal context 
only some one or several will predominate and be the critical training ' 



IC. f - . ' iii-ior , 



In 



. fAEG REPORT N0.» 12-3 ' ' " ' , */ 

. ■ * . 'task subtask "page 

• ■ ' 05 43 ' 02' 

Each of these functions has a learning and trsining requirement that' 
differs somewhat from each of the other functions .. , , 

It is.poss'ible for learning to-. be achieved in each of these task functions 
independent of learrring on other task functions, and at suitat e stage 
ifn??;"^^ effectivel-y^integrated with what has been learned indepen- 

.dently on other task functions or elements . This ability to combine part- 
1 earning. efficrently into whole patterns of ability does'not apply to'^anv 
randomly conceived sybset'ting of behavior and performance~^lements". ^ 

this subtask, the set of task structure elements of major'util ity wi 1 1 * 
merely be listed, and brief identifying descriptions will be given. The two 
^ e-jceptions are- Procedure.Following and Tracking. T^Appendix A, most' of 
these terms are given a liflore detailed and analytic treatment, and implications 
^ f 95 training will be spelled out. The assessor is advised to become familiar 
WT^h task structure in the present context because of the power it can confer 
-on the design, of, highly efficient and c6rt-sensi>tive systems of overall skill 
•training,. , • 

The list of task structure elements consists of: 
Goal Projection ; 

Scan-Detect ' ' ^ \, ' 

. • Identify ' \ 

Interpret . 
' Procedure Following 

Decide • ' ^ y ' ' , ^ > ^ 

Construct/Plan « . * 

Tracking . . 

- Motor Performance 

Interpersonal Interaction 

Recall Task-Cycld Information 

Recall Enabling Information. ^ . 
• * ' ' { ■ 

These elements are briefly defined qn the following pages under Procedure SteD 1 
Most of them are treated-more extensively in Appendix A. T ^^^P ' 



PROCEDURE ' . / * 



Hnn .-n [h •'Ji'^Vru*^^ descr ption of the target tasks in the innova- ' 
tion in the light of the definitions of the tas\ struct-uVe elements given 

?o ?he'?nno5a?iSn ^ ' ' ''^ °' "^''^ ^''"^ ^l^ments.cwhich appl^ 

* • 

, given perforSnce Linchi-eJe t hnulH?'" "t-'^'^y "^^•^''t^^ °f » 
V terns of a competentberfomer aitl^n tl^ i.t realisttfc and practical in 

^ . , : The ,oa, projection sfr:r?:-i^?t']a"?e1:l^"J-fo?! dlJetl'^nVSIi^fr ' 
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to a work cycle conclusion. A test of the validity of the operator's goal 
projectioi), other than in his performing the task itself, is his ability to 
discriminate an adequate from inadeauate representations of proper goal 
states^ given a concrete set of conditions under which the task is to be 
performed. 'If the operator has a clear and complete goal image, it is 
possible far him ta compare the results of his actual perfonnance with the 
results of, criterion performance, note the deviations and thus give him- 
self "learning feedback" which he can use to improve his performance when 
he perfg^s the task again. The goal image thus serves the dual purposes 
. of di reacting performance and of directing effort in skill improvement, 

• Scan-Detect ; Scanning a work environment, using appropriate senses, and 
detecting the presence- of cues, or conditions requiring some form of action 
to be taken. The action may consist of further examination of the cue, 
leading to an identification or to assisting an interpretation ; or an 
inmediate overt action (such as avoiding collision wi th an obstacle) may 

be tfie result. Learning to scan-detect in a given task context is learning 
what to pay attention to and notice; in complex tasks it will consist of^ 
suitable divisions, of attention among sources of cues. In some tasks, 
scanning technique may require sophisticated search strategies, or emphasize 
sustained vigilance. There are tasks in which the detecting component is 
made difficult because relevant cues are obscured by noise, irrelevance or 
distractions. * 

• Identify : To differentiate an object or process or condition from others 
in a set, and generally put a name or label to it. Or, conversely, given 
the name, to point to the applicable object or condition. To identify may 
be not only to differentiate, a$ one hamijier from another, but to generalize 
within a class or set of entities such as "hanfners". Being able to apply 
verbalized job instructions demands, the incumbent's ability to link the 
appearaace of the object (or the making of the operation)^ to the matching 

. word or expression,. 'In some tasks, such as target identiVi cation in radar 
displays or in sonar presentations^ the skill itself is in the perceptual 
process of .detecting and organizing cues that, as a pattern, will yield a 
recognition of a^n identity. 

• Interpret ; Responding to one or a group of cue$ as a pattern or context 
that has meaning where the meaning Is a cognitive inference. The letter 
symbols "cat" will tend to have a different signtficance or meaning, 
depending on whether they occur in a discussion about domestic animals, 
earth-moving equipment, or malicious women. Interpretation requires adding 
information to that which is physically presented as * individual cues. It 
may also require responding to a collection of separate cues as a. pattern 
or gestalt. Interpretations may be Incorrect: a power cord out of the 
power source socket may "m6an" that the appliance is not in operation 

but only if the appliance lacks some other power source. The output 
actiorrs in^some tasks may depend heavily on the incumbent's ability to 
respond to patterns of cues in terms of what they reference or mean 
rather than to their individual presence or absence. In perceptual tasks, 
..an example is reconnaissance photo interpretation." In conceptual, cognitive 
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tasks, an example is tactical or strategic planning on the partial basis of 
signs by the enen^y that are interpreted as his "intentions". 

Procedure Following : Carrying out a sequence of discrete actions or steps 
in a more or less invariant order except at conditional branch points. The 
sequence of steps in some classes of procedure is more fixed by prescription 
than in others. The procedure is ordinarily initiated by an explicit demand 
signal in a work context and terminated by a specified final action step 
or identification of a goal state. Each procedural step is characterized 
by: (a) a stimulus state or specified cue appearing on one or more indi- 
cators or physical objects; (b) a specified task action; or (c) a result 
of the task- action appearing as a specified cue of response adequacy or 
inadequacy; this feedback cue may be the stimulus for taking (or selecting) 
the next step in the procedure, ^ . 

The set of steps to be taken in a radar checkout or alignment operation 
constitute a procedure. This is an example of a highly prescriptive pro- - 
cedure; each step is well-marked, the sequence is fixed, the cue for each 
step is explicit and unambiguous. (The exceptions may require an 
"interpret'' task element,) 

On the other hand, the set of steps to be taken by a surface vessel in , 
locating the position of a suspected enen\y submarine may be a highly ^ 
strategic procedure: although a principle specifies what general steps 
are to be taken and in what order-, each btep is largely conditional on 
environmental context and on the results of previous steps. More informa- 
tion processing is required to perform strategic procedLirss than to per- 
form prescriptive procedures, therefore, the latter require less training 
time and facility where other factors are held equal. Training for strategic 
procedures overlaps considerations for decision-making training. 

Special recognition needs to be made about activities loosely called 
"Communicating Verbal Information," In some work settings, the structure 
of the communications is highly standardized. One example is that of the 
control tower personnel interacting with pilots. Another is that of the 
mechanic filling out a maintenance form detailing the* work he performed. 
These are examples of procedural formats, and may be taught and learned as 
procedures. Remember that learning a procedure should also extend to learn- 
ing to cope with contingencies associated with the procedure. A second 
class of "communicating verbal information" requires the more or less free 
form construction of a narrative or scenario: the technician's explanation 
of "why the flying of a mission with an airplane having a given defect woul.d 
be dangerous" would be an example. The task element rubric '"Construct" would 
apply to the latter as an indicator of training and of perfonnance character- 
istics. But where the rational and objective factors must be supplemented by 
skills In perceiving, and formulating responses to emotional hnd attitudinal 
factors in interpersonal communication, a new dimension in task requiremsnt, 
aptitude, training and performance is introduced. This dimension is 
acknowledged in the separate task element called "Interpersonal Interaction." 
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Procedures may combine prescriptive and strategic components. The improviseds 
welding of hull pTates at sea would provide many examples. Using heavily 
damaged engin{?s to bring a ship into port would inevitably combine the pres- 
criptive with tha strategic. 

Learning to execute a procedure Is generally preceded by cognitive phases in 
which the component stimulus indicators and indications are identified and 
located and named, as well as the objects (controls) on which the student's 
response is to be effected. This enables the student to use verbal or 
diagrammatic information to forinulate a conceptual model (or picture in the 
head) of the sequence of cues and actions mak1;ig up the procedure. This con- 
ceptual picture, if it is sufficiently clear an4 the student-is- able to trans- 
late it into perceptions and actions, can guide student performance. This 
cognitive image may bs supplemented during practice by job manuals that de- 
scribe the procedure. 

Retention and recall of a procedure is poorest when the procedure has been 
learned to an intermediate degree. This is the level at which the student 
may have learned many of the behavior components, but no*: their relation- 
ships and sequences, either cognitively or in perceotual -motor behaviors 
mis level may be evidenced by an appearance of "bare mastery" of the pro- - 
cedure such as in an early, stumbling carry- through of the procedural 

^^^^ ^^'^'"^^ training for many students terminates 
at about this point in. learning level, and further practice in the task 
resumes on the job itself where procedural errors may be difficult to identify 
?«nHc detention, as measured by recall and savings in relearninq. 

?n?oL ; "^""l sufficient repetitive practice! has 

integrated the procedural components into a smooth sequencing of perceptions 

At this stage, he can be intevrupted an'd then resume readily 
where he left off; he can anticipate what step he will be'takinq next- he 
can retain information specific to a given task cycle. In this^sJage: he - 
^ rnIffvS^n/°^ merely a chain of otherwise isolat-d behavior elements, but 
a^™FfeLirS?lSe?cIs'''°'^ ' ^^'^^ination, resists forgetting 

Decide: Selecting a course of action by systematic processing of informa- 
tion about outcome requirements and the situation on the one hand, and 
response alternatives and strategies on the other. Choices are usually 
made in some degree of uncertainty about the actual situation, the need, 
and how a given choice will work out. For example, under marginal weather 
conditions and a given importance of mission and a given aircraft with per- 
formance limitations, to choose whether to fly tha mission or not, assuming 
an option for choice exists. In general,' a decision differs from a straight- 
forward procedure in that all of the information necessary and sufficient 
for choosing an action is not available, or is ambiguous, so that probability 
and risk (in some degree) are involved in selecting an action. 

Note that where specific' rules cover the selection of the "correct" response 
from potential alternatives, the operator learns a procedure. The procedure 
may be complex, but is nonetneless a procedure and may be learned by the 
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mechanics of procedure learning. There may be occasional gray areas 
between a task that is performed as a procedure or that may be per- 
formed as a decision-making (or "judgmental") operation. Even in the 
latter cases, however, the separation of the procedural format from 
the decision-making format for managing uncertainty may be useful for 
instructional and performance purposes. 

The greater the number of variables that should be considered in 
reaching a choice, the more difficult it is to learn to make the deci- 
sion proficiently. The multiple variables may characterize the 
definition of the problem situation, or multiple options may characterize 
the choice of the response, or both the situation and the resources for 
response selection may be complex. Because the selection of the response 
is made on a probabilistic basis, it is generally impossible to be " 
absolutely rigorous i n •evaluating the student's ctrotce, except where an 
ineffective choice occurred because of neglecting a key variable in the 
situation or a key variable^in the resource committed to a response. 

The decision maker learns to cope with the variables in the problem 
class; he must also learn to cojDe with the kinds and levels of un- 
certainty inherent to the class of decision he makes. 

. Construct/Plan : Creating a response pattern that tends to be unique to 
a unique pattern of conditions and, objectives and resources. But the 
variables in the kind of construction that is made (such as a table of 
organization) or in a plan (such as planning .a landing operation), tend 
to be constant from one case to another. (The variables in planning a 
landing operation tend" to be about the same from one landing to another.) 

Tracking : Responding to a more cr Ic-ss continuously changing stimulus 
by more or less continuous changes in response action , having the critical 
requirements of smoothness, coordination and timing. A given tracking 
skill is the manifestation of the operator's anticipation of the dynamic 
relatipnship between targ6"t data (the movement of the "cursor") and given 
actions on the controls he manipulates. In pursuit tracking, such as 
flying, an aircraft visually towards a moving or stationary target, the- 
operator can differentiate the dynamics of the target from the dynamics 
of the device he, operates with controls (his aircraft). In compensa tory 
tracking, such as in blind flying to the aircraft's attitude Indicator, • 
the dynamics of the target (such as air turbulence) are not readily 
differentiable 1n the displayed Information from the dynamics of the 
control instrument (the aircraft's responding to control njanipulations) . 
This confounding of dynamic characteristics of the external target with 
the dynamic characteristics of the controlled instrument or vehicle 
makes learning to anticipate their interaction very difficult; compensa- 
tory trjicking is notoriously more difficult to learn than pursuit tracking 

Piloting^ a vehicle differs from aiming a gun at a target. A vehicle must 
be moved through apertures that the pilot must project ahead, to the side, 
and even to the rear if he is concerned with safety. The aperture must 
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be large enough to include tolerances for the pilot's skill, for 
probabilities of invasion of the projected aperture by obstacles, 
either from the sides or on the surface of the roadway. Maneuvering 
a vessel has equivalent variables and requirements. The skill of the 
operator depends as much on his ability to project the behavior of 
aperatures in his operating environment, as on his ability to project 
the behavior of the vehicle in response to the control dynamics he 
applies to accelerator, brake, and, steering wheel. 

The technology of instruction and learning of tracking tasks is an ex- 
tensive literature not to be reviewed here. There is little that can 
be learned from verbal materials that transfer to tracking tasks 
Tracking must be learned by doing. On the other hand, there are many 
examples of sirnul ators^ tjia:L seek levels of physical "fidPljty" in dis- - 
play-control dynamics that are unwarranted logically and psychologically, 
but achieved at very high cost. ^' j' s jy 

In real -life tasks, the operator must generally attend to multiple 
streams of information a|: the same time. To do this effectively requires 
large amounts of practice. The performance of procedures is often mixed 
wiuh tracking activities. To the extent that the procedure does not 
depend on information contexts outside the procedure itself, these oro^ 
cedures may be efficiently learned tojiigh degrees of mastery on pro- 
cedural trainers before the student practices them in full work context - 
that IS, on a high-fidel ity simulator where the operator must make suit- 
able divisions cf his attention. •. 

It should^ be clear that an _ understanding cf the learninq process enables 
the Tormulation of -a training schedule or scenario which maximizes the 
learning and transfer of training benefits from a family of instructional 
devices, each of which may cost an order of magnitude less than the next 
higher in functional capability. This issue relates both to Stage of 
Learning and to part- task training considerations. 

Motor .Performance: Executing a pattern or sequence of motor activity 
wnere the smphasis on the totality and its parts is on dexterity, control 
strength and precision cf output. The major concern is on differentia- 
tion and organization of muscle groups into performance patterns 
Examples:>s> typing and other keying operations, hand-to-hand combat, row- 
ing a boat, sports m general ■ In a widely different context, control 
Of the voice in speaking, singing or in learning to pronounce in a 
toreign language. 

Interpersonal Interaction: Apprehending and presenting information to 

H^n K Jl''"^"'!' ' '"'""'^ ^^'"'^ facilitates getting a collective job 
done by the participants. This task function connotes more than pro- 
cedural factors in the sense of job manual instruction. It includes 
personality elements, attitudes, interpersoiul predispositions and 
style -- ana other concepts that are difficult to reduce to operational 
definition except indirectly, sbch as by the observation of 'performance 
consequences. It is outside the scope of the present project to develop 
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analytic structures and rationales that apply to the learning and ex- 
pression of interpersonal skills. It is relevant, however, to' 
acknowledge them as a special and important class of activities 1n anv 
work environment, and to recognize that in some jobs they may be as 
mportant than technical competences: intimate leader-follower re- 
lationships and team coordination situations are examples. 

• "^p*^-"/--^ Info^Aion: Remembering and applying data peculiar 

to a work eye e and rel^v^to the activities performed in that work 
cycle. Examples: remembering what one has already done and not done 
" cSndit'S"?.,^!; I ^"^f^^l^^^i a.Pre-mission briefing; a special operating 
condition such as a device having an intermittent. fault; the temporary 

process of being aware of 

Y'^^ 5^^ . cofflpleten ^anxi Jiol completed wWle performing a procedure 

Tn?^!!i%?" function. It is necessary to..hold in mind the 

Tn nc^J ?" over time that enters into the making of a decision 

In psychological jargon, tnis task function may be called "working 

fnJ^rL ? "lemory^ Its operation is obviously essential to the per- 
ti^f" ?r ^"^ 'It"^ i^ost primitive conditioned response 

■ L^hV.l: Cl^ai^ly there are training and learning techniques that can 
ReliSMp°;;:.,°?i*''? J^'^r ^^^fd.. depress the effectiveness of this function 
Reliable recall of task cycle information depends on the learnina and 
exercise of task formats as described in Taslc 0^. Subtask 23 ^ 

• Recall Enabli ng Informat ion: Remembering and applying facts rulG«; and 
. |°^^!Pj^f^ to taskVrforSance ' The classes o? 

. JnlSw ^f^^ have been previously identified as Enabling 

Knowledges Enabling knowledge ranges from the stall speed of the air- 
^h^J?;'n2"^J' Centigrade equivalent of 70 degrees Fahren- 

hett or the shank diameter of a number six wood screw. Correctly 
' rnnH I l^""^ information in response to a classroom test is a different 
nnSpi Iho performance from that of recalling the same information 
under the stress of operational conditions. In the Appendix A treatment 
Of this topic, the concept of this long-term memory is called "S to?? 
in order to distinguish it from "Short Term Memory". It should be 
recognized that the perfonflance of Procedures implicitly implies the 
rau?I\h°I "f^"^"^^^^^"' that supports procedural performance ^ But be- 
■ can haJe ai??S2^:nJT'^'-'"9 and applying of support information 
?nn Jf! aptitude and training requirements that differ from the learn- 

tion justifies its own identity and ^;haracterization as a task function 

Furthermore, if two tasks differ in procedi-l operations, b t demand 

c ,r!l'V^?"" °^ enabling and support Information, there should be ^ 

Iftll'fk'^ ^'^■"^■"S f^^"' to the other that is evident 

after the early stages of procedural learning in the second task 

^' con^lxt^^hnurfLf.J^! relBs^ant task structure elements and any information 
^an^r^f T^*"'* ^'^^^ stages of determining 

range-of-effect. In many cases, this subtask of selecting task structure 

" id , 
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element(s) will merge Into the Subtask 05.14, Identifying relevant jobs 
and tasks in the Navy Inventory of jobs and tasks. 

3. Proceed to Subtask 05.23 to determine If the Innovation Is specifically 
applicable and restricted training in task formats. 
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TASK: DETERMINE RANGE-OF-EFFECT 



\ 



SUBTASK: IDENTIFY RELEVANT JOBS AND TASKS 



DESCRIPTION 



I^hiP^L ^^.•''^ in deterrmmng the range of the innovation's effect will be 
ach eved in this subtask, assumins that training courses will be readily den- 
tified when the specific job tasks have been identified. In all cases the 

to'desrr?hpT'r ^" ^^^^1^^^/^° think beyond the surface match g o" word used 
to describe tasks and attempt, rather, to match on the processes and thinas that 
•• ie behind the words used to describe them. "Disassemb ing" ocket w tc a d 

• re Dec??fe^Jk"? 1? ^^T..'"''^'": ""'l^''^ '^'^^ °f overlap^-n ter^ of ? eir 

' "Sri^^^^ operations for acquiring them, but . 

nftH ^i:^ likely to be greater than their similarities. 0^ the ^ 

^nnS' in the names- of objects or of tasks are good hypotheses 

for looking further into the relevant similarities. 'lyijuuicbeb 

Finally, however the decision of relevance or non-relevance of the innovation 

Sdamenr''Fn.'p5 '''^ °' '^"^"^^ °J ^""^ ^" °f personal ns h nd 

judgment. For example, an innovation in instruction on aiming and firing hand- 

?inn ^'n .^P^^hown a 40% efficiency advantage over other methods of inst?uc- 
. .?J;n..?r^ ^PP^^ t° a'i'ning and firing of rifles? Of 

elSte^hrnL']^TP°^ °' ^? S^n^^^lized list can osslbly 

volveS in this decis?on!°'' °P^^^^^^°nal knowledge about the various'tasks in- 

It is realistic to assume that the empirical testing of the innovation will 

2?rl'U'''r/' 'fP^' P°^^"^^"^l ^^"9' °f traim g prob em to 

which It might apply. Therefore, , the potential value of the innovation must 
be supported by informed guesswork. The more "similar" the candidate !ask is 

s?m t' b^n^;' t'l'' I'l '''''' '''' candfSate'Lsk willlhow^ 

similar benefit But, what may seem similar on external appearances mav 
frequently not be similar psychologically, and vice versa 

^ KnJ?J^H'"^-?-'?li"" therefore proposed. The assessor has already 

broadly classified the relevant job task into one of the five qroupinas- track 
1 g; procedural; decision-making; constructing; or leadership-?n ePJe? onal 
t nf th' ?'°"P^;?' "'^^ ^''^ f'^^ther classified the instructional ob jec- -^^ 
tive of the in/Tovation into one or more task structure elements- qoal-imaae 

ranSe oT^ni; t fJ'^'^^' '° generalize and dLh.U the re e In t 

ItJ M l-^ °^ ' combination of the concrete identifying factors 

that define real tasks in real environments that will be listed below 

larilbies'^'Ihp'H.^f.^''^'""?""* variables should be treated as coordinate 
variables; the data they develop are a complete pattern In some rases thP 
description of the innovation will point rather directly to the Variable that 
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is primary for generalizing range-of-effect. For example, "training in use of 
handheld soldering iron or soldering gun", identifies a procedure for using a 
tool, and all job tasks that call for soldering iron skills are candidates. One 
could search a data base of descriptions of Navy jobs directly with just 'that 
information.. But an innovation claiming efficient instruction in "strategies 
for operating vehicles through turbulence" could, potentially , apply not only 
to aircraft (on which the jnnovatior> may have been validated} , but also to 
surface and sabsurface vehicles, as well as land vehicles operated over rough^ 
terrain. In this example, the key factor in stipulating a potential range-of- 
effect is "operating a vehicle through turbulence", so the search should look 
for the operations and the vehicles (equipments) operated in an environment of 
"turbulence". The linking term is turbulence. 

These examples should demonstrate why it is not practicable to set up a limited 
set of "taxonomic descriptors with universal range and applicability to job 
training and performance. 

The following variables, taken together, comprise a job task context that^nust 
necessarily be regarded as guides, both in probing for the potential full- 
range of the innovation, as well as for delimiting that range. Nor should the 
order in which the variables appear determine the ordet in which descriptive 
terms are sought; the order should be based on the key characterizations in the 
description and understanding of the innovation. 

The objective variables that characterize task contexts are: 

Equipment and objects used in the task 

Environments in which the task is performed 

Toclc used in connection with the task 

Reference information used in performing the task 

Task operations (made up of "task structure elements") 

Criteria applied to task performance such as "time" and kinds of "errors". 

PROCEDURE 

1. If applicable, classify the training^bbTec'fTve in terms of target job tasks 
into one or more of the following categories: 

Operations 
Maintenance 
Service 
Command. 

This is a broad grouping, but it may help as a rough first pass at subsetting 
the inventory of Navy jobs into a target group. It should be noted that an 
efficient search strategy is one which, as early as possible, excludes the 
largest segment of the total universe to be searched as irrelevant to further 
Inquiry. This is the Twenty Questions Game principle. If tKe innovation 
cuts across any kind of categorization such as suggested In this step, 
abandon it. In any event, go on to step 2' 
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s1c;ion%^al'^fn?in'^'^H''^^*^'V'' "''""^ described in the topical 
sections that follow, identify and set down the objective or external 

a.n^l'.^h'vf °l ^''^ structure elements JhafHrn? the 

applicability ofv the innovation. For example, the use of an electr c arc ■ 

c^ re' Z'if'.'^' i°i^^^ '''''' Characterizes as: P - 

ceaure -- Use of Arc Welder. But a turn on procedure specific to an ara toy 

meSi'^S'the' U ''''''i''' ^° the'operator ort^I^AB^-T e 

ment and the field maintenance job on the ABA-lOX equipment. 

If there is some doubt about whether to use a more restrictive or a less 
ch a' terl-att^^^ 't' '''' restrictive, or more genera 

for inH^IInfc !' now being constructed is a search of candidates 

the ob^?a k HpIn.'? H I ^«^^«,^°r will make another examination o? 
the job task descriptions in their individual contexts, so he will have an 
seaS"^;p%^? reject those that do not apply. On the other ha ? ?he 
search spec fication is more restrictive than it should be, candidates that 

elrcT'^B: ^er" '''"'.''.h''' I'r'''"' '''' fn tJe re r val 

search. Better to cast the net too wide than too narrow. 

DlSrnc*Ifc^°^*-i"'""^ P^-^P^ "lay be established on the basis of 
patterns of similarities among the following characteristics: 

• Equipment and Other Operational Objects Used 

The name of the equipment or class of equipment operated by the operator 
or maintained by maintenance personnel , 'is usual 1> a primaVy desSor 
for Identifying jobs and, by implication, of the Lks^aLodatL^ith 
^eSSioment'naniL thp H^Jr'f''^" °^ innovation can include relevant - 
o3 JhP nJhpf h^^H 5^'^ narrowed rather quickly. 

fn I-^??"^' innovation extends to equipment in addition 

to that initially specified, the greatest penalty in restricting the 
proper range of the innovation will be imposed. ^^^Lricting the 

In assessment practice, it will be good policy to identify equipment 
names and types after task structure ele^nts and relevant env ?onments 
have been identifTi^ unless the innovation has to do wfth mn p2 at ve 
s e-'i-^'-NnVriE" activities tend to be equipment and ISol 

speuiiic. Note that, tnis recomme>idation is embedded in the oresent orn 
cedure - see Subtask 05.13 on Task Structure Elements ^ ^ 

Environments 

An environment sets.condi tions under which a task is to be performed 
The differences in conditions may imply a different kind or level of" ' 
skill. Aiming a gun from a stable platform is different from aiminn a 
gun from a moving platform. Driving a vehicle on frt ter?a?n if?ers 
.^'nn VV^.^ °? mountainous terrain, and both differ from dr Sinq in 
congested downtown traffic. Perfomance in forgiving environments 



133 111-112 



TAEG REPORT NO. 12-3 



TASK SUBTASK PAGE 
05 14 04 



such as stall recovery at a high altitude, may require a skill that 
differs both qualitatively and quantitatively from performance of the 
same task in an unforgiving environment, such as stall recovery at a 
very low altitude. 

Tools - 

Tools may be regarded as a special kind of equipment. The ability to 
use a given tool, such as a recording voltmeter or a torque wrench, 
implies some general range of tasks and skills, but must usually be 
qualified by the equipment or materials or other work objects appli- 
cable to the task context. It is possible, however, for the name of 
a tool in itself, to constitute a search specification to a job task 
data base. 

Reference Information 

Reference information data that may be in the incumbent's head, 
available externally (such as manuals of instruction or handbook^ 
in normal, expected conditions of performing the task which may be^ 
essential to carrying out the task. Examples are the normal waveshape 
for a given electronic checkpoint, the number of U.S. gallons to^irr 
imperial gallon lo an American motorist in Great Britain, the tempera- 
ture at which a gi^/en flux will bond' given materials together in the 
task of welding. 

Reference information may also consist of procedure information to 
supplement or substitute for memorized proceclures. Such supplements 
may be necessary for procedures that are performed relatively rarel/ 
by the incumbent. 

Task Operations 

Responses made by the incumbent t^ task situations in carrying out his 
duties are denoted here. The descr'iption of task operations may be 
made at many levels. Task actions can be described in operational terms 
with task Structure Elements as a general format for scenarios of pro- 
cedures, decision-making, constructions, and so on. 

Task characterization^^ are generally weakest in the descriptions of 
perceptual-^cogni tive demands on the incumbent. However, these demands 
are often the most important for skill learning and for the' transfer of 
skill from one work instance or situation to' another. Scenarios of 
task operations frequently tend to emphasize the action demand of the 
moment, while neglecting the concurrent range of perceptual-cognitive^ • 
activities that differentiate the mechanical step-by-step prescriptions 
For action from fffe full context of action required for competence. 
Thus, safety and efficiency in driving in heavy traffic require not 
only steering the vehicle in order to avoid collisions, but continuous 
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awareness of the dynamic changes in the size of the "slot" in the 
traffic w-hich constrains the driver's maneuverability. 

• Task Criteria \ 

These are the variables in performance which apply to determinations 
as to whether the task is performed effecH^/ely or ineffectively, and 
at what level of effectiveness. Some ultimate criteria may be imposed 
by nature, such as in the recovery from a spin in an aircraft or by 
the enemy in a combat situation. Secondary criteria may be 'set up by 
A instructors and evaluation experts who observe the extent to- which the 
> actual operations canform to prescribed operations. 

General criterion variables consist of time to perform tasks or task 
actions and kind and frequency of errors or deviations from prescribed 
norms. A third criterion, less explicitly applied, may be the quali- 
tative and quantitative range of s.ituations with which the incumbent 
may be stressed, before performance breakdown or failure. Increases 'in 
situational complexity would be an example of such stress. 

Add the job task characterizations derived from steps 1 and 2 to those 
obtained from Subtask 05.13, Select Task Structure Element or Task "Fu 

Organize. the descriptors into' one or a set of search arguments'with appro- 
^""^^ MOT ^^^^ connectors. A logical connector is either an AND or an OR, 

or 3 INU 1 . 

For example: * ' * 

Type of skill training; Construction 

\ 

■ Task structure element: Procedure/Manipulation ' . ' 

Task characterized by: • , " 

Environment; Improvised in field 

Objects used: Softwood OR hardwood, NOT metal 

Tools: Han'd saws AND improvised bench 

Opera t.i on: Sawing 

Criteria: Not appl icable. 

(The example is artificially simplified.) '' ' \ 

Search the Navy inventory of job task descriptions for those relevant to 
the target kriowledge and skills in the innovation. Step 4; just preceding 
this step, will have yielded one or a group of search arguments, including 
descriptors and logical connectors between descriptors. It is expected 
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that the descriptors- of Jobs and job.tasks in the Navy inventory will'have . 
matching-'descriptors for automatic search.^ If this is not 'the case, hurnan 
interpretations will be required for-. comparing the innovation descriptors • 
witih the job inventory .descy^iptor/ and deciding whether the concepts be- 
hind the respective wor'^ls used aire, for training purposes, a match or a 
mismatch. • 

Bear in mind that the final decision as to whether a given job task in the 
Navy repertory is or is.jnot relevant to the innovation, tvill be an inspec- 
tion of the context and 'tests of the meaning 0/ the clescrip^tors of the Navy 
job task;^ That is,^ the hits made by the search operation will not be 
automatically, accepted as relevant, but'only'as candidates for acceptance. 
This is similar to inaRiing a search request for a book in a library; the 
search by the librarian results iw a number of books*, each of which is 
examined before making a final decision to accept (^i?>eject the candidate. 

Seardh strategy has trade-offs. A small set of relatively abstract descrip- 
tors/will tend to result^ih-a large number of hits or matches with items 
in the listings, but a large proportic^n df these hits, upon- fuller examina- 
tion of their content, may turn out to 'be irrejevant. However, the- proba- 
bility of getting a large proportion oT re^Ievant items will be high. On 
the/ other hand, a large number of specific and qualifying descriptors will 
tena to result in Si smaller number of 'hits in the-listtng, arid a larger 
praportion of these hits will turn out ta be relevant. 'However., the risk 
is /increased that many potentially relevant candidates wil l be missed in 
the search. . ^ 

T . / . 

Injcommdn practic^, important searches are made iteratively. The first 
nejft is cast widely and its contents sampled. If the sample reveals a 
lalrge proportion of rejects on the basis of judged irrelevance, additional 
qualifying descriptors arjgj^^^d to. the search specification and the search,- 
i 9 rerun with a consequently smaller number of hits. 

Much the same search principles may be applied when' inquiry is made into 
the knowledge of the expert on Navy job tasks:^ He will formulate an' "image" 
of the job ^sk characterization of relevance and evoke images from his- 
memory that tend to match. The s-earch may be facilitated if the' searcheV 
scans a list — in thi^s case, of the names of jobs and, perhaps, dut'les 
within jobs in' the Na^y Inventory of jobs and duties. 

An alternative to a search on job and task listings and descriptions is to 
search on course names and course objectives. This choice assumes that the 
innovation is relevant only to those skills/and skill /tontexts for which 
courses are available. This operatiorv^is afjump tojSlibtask 05.16, Identify 
Relevant Courses. ^ ^ 



The result of the search and verification of }0tr tasks relevant to the effect 
of the innovation on training and/or performance benefits, is a list of Navy 
jobs ^nd job tasks (or skillsj and such qualifying information aS'- can be 
added to each item on the list. ^ ^ . 



ERIC 



III^lTS 



T'AEG REPORT NO. 12-3 



-J 



TASK SUBTASK *PAGE 
- 05- 14 07. 



Fqllowing the identifica-tion of the Stage of LeaVninq tovwhich the innnw;, 

• The number of existing incumbents. 

' °J number 'of studehts who 'are to bff trained for the 

. • job and job task in each of the next five years. • ' 

.,• Any expectations projected about change's in the job or-iob task 
structure and when they may happen in thfe five year ?uVu?e- -- as 
' "jSrincumSenr'^ °" ''^^"^''^ innovation for the 

* Dnmi?.'Hnn%°^ changes"in the charac;terTstics of the input s-fudent " ' 
the ]nnivatfon?°"''" *° assignment in the jobs'relevant to 

7. Determine if the desvipti'on of the job in the Navy job inventorv inrli.riA 

k o" : ge" jriwi^ iTso""? r?." ^1''"'^' '"^ ac ""nT^hr S'" 

A J • Information- and, if it is sufficientlv 

8. Organize and retain al'l data obtained in this subtask Tt'un-n ho n^r..^- 
o^S In'd^^^rja'LV" ' i^alNlt;iS:to'?hrJ^v^n™Sa^™'e 
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TASK: 



DETERMINE RANGE-OF-EFFECT 



SUBTASK: - SELECT RELEVANT STAGE OF LEARNING 



^DESCRIPTION 



Different stages of learning call for different factors in training. An innova- 
■ tjon that may be effective for one level of skill acquisition may be irrelevant 
or even counter-productive to another level of skill. Differences in skill 
level training may be reflected in course levels, but more specifically in* 
units of training within a course. The characterization of skill level, there- 
J- fore, enables a more precise determination of the training operations to which, 
the innovation is specifically applicable. 

The practical aspects of stages of learning are those of acquiring a skill. 
The reason for concern with stages of learning is that the minimum fidelity to 
the job task for effective progress in skill acquisition changes from one stage 
and level to another. Whereas words, diagrams, and pictures may be adequate 
and effective at an- early stage in acquiring a skill, or the components of 
skilled performance, later stages demand more physical and functional fidelity 
for useful growth and' transfer of skill from training to operations. And where- 
as, in early phases of skill learning, adequate student response may be verbal, 
in later stages the student response must be the literal equivalent of the job 
task response with the physical work objects and tools of the job. Temporal 
and other dynamic relationships must become realistic so that .the student 
develops proper cause-effect "anticipations" of the tool, equipment, material, 
and environment. At/high stages of skill learning, the full information- 
handling load of simulated or real environmental conditions and contingencies 
must be presented, else the skill will level off at less than a reliable, on- 
the-job competence. Not only must higher degrees of physical fidelity to the 
real job task accompany higher stages of task learning, biJt the form of measure- 
ment and analysis of student performance changes, as do the desirable properties 
of instructional control. 

• Sophisticated awareness of cost-effective instruction will take into account the 
minimum requirement for students', consistent with effective learning at-a given 
stage, and for the transfer^ of that learning to a later stage of what is being 
practiced and learned. For example, a growing body of studies is revealing the 
fact that many sequential procedures can be learned to near-operational levels 
of proficiency on relatively crude mockups, y^ather than on operational equipment 
or expensive, high-fidelity simulators. This section of training analysis, 
therefore, characterizes the mapjor stages of skill learning at which various 
kinds of instructional options; with substantia! implications for cost-effect- 
iveness, can occur. \ 



A summary of stages of learning emphasizes the student's learning processes. 
With continued practice, the behavior processes within the student tend to 
change. These changes are more or less systematic and predictable, arid depend 
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^nn'ovJlnJ*'^"'^ sequencing, kind of presentation of content, and the nature 
p?ope? ro^^L's^'En'?'^ P^"» here, refers to an fa? a 

than 2Znf aT^^^ "''"^ °^ ' Transitions are gradual 

ratner than abrupt, and fo.r a given student not all aspects of a task hpinn 
learned make the same shift at the same time. The niajSr rransition is frnm 

^ mo?8 nTo™at?on , Jh» ^^<«^)becmes increased so that he can process 
afTS^"^=°"" 

t s CM !; This. automatization of behavior tends to develop in Eercepiual 
^in^h^Z^^l^T'- organization Of .™otor^cti?ities^:"^' 

What tends to get automated are the rel-ative constants among a set of task ' 

ex c;S"or^;Hp''^^■^P''^"■^^'^' °^ °^ ^^e mo e, ent pa ns 

executed, or the relationships among all three. These constants in terms nf 

"tasr?oma "''T"'; T"^ ' °' experiences"L7be 'tflSugh^ f°Is 
cessina 1?nT n; / u '^"J '"'^ °^ ' ^^riah^e in a task input, pro- " 

cessing ink, or task output. An example is "altitude" in flvina conceotu^nv 

aHiL%P ;r '\''''t^.^y flying the aircraft, the m^men I ^MfSff^^e 
aftimeter is the value which th s variable may tffke nthpr v3H;,hipc hnVh 

^^?:^r'^l^'r^ heavier-than-air fl?^ht'\'re cSJ^se^ h'a" ^ de^r 
Tctuld ?Snc2;nf K'"^^T*'°? processing format of necessary variables 
(called constants above) for flying includes, but is not necessarily restricted 

task rac?icp''.hfn''!' 1''''' internalized this format of var abl V xt p'^^ 
task practice, the. pilot can more readily use different ofivsical sourr'prnf ' 

th7^re1ne1p:r?:n^S^S^^ilrd:L':fo% 

/rLlrjs' io%=r^-„%-^^Tf^a^^a^s - 

«^n1"'::ch l^lj;^^;^^ decisionsl^Inrtr^SLIru'r .. 

i's'es,°enHal°L'"'' ^"Tl^' fH"' P™""" "hereby task foraats are required • 

i^^'H^aSe^?ilSS!r:^^^^^^ 

approxL?i:n"::y'be\ e'q te reuSr^hl ntenl'^' h'"' *^ 

tions, course element objectives. ]Ssl?5c"o;"Sa;J:,s:*:Sd' llTe^/tV.llr- 

PROCEDURE 
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Orientation and Familiarization 

These may be general reference and context knowledges. They may include 
information about goal states, environments, appearance and properties 
of tools 'and other resources, commufii cations and job interfaces, roles 
on the job and roles in the training environment, and relationships of- 
tasks to each other. If these kinds of infoi*mation are not presented 
and credible to the "student, he will tend to develop his own images 

.and hypotheses. The concepts acqui red- during orientation may, in some 
cases, serve the. student as mnemonic models for learning, retention, 
and generalization — that is, he acquires a general .picture of what to 
expect and the more specific content he learns gets ti-ed 'in with these 
expectations. The expectations can serve to organize .the details of 

. what is learned later. 

Theory of operation is "included here because it may serve mainly as 
orientation to the skill, rather than as a component of the skill. It 
becomes a skill com,ponent only if it, becomes integrated with practice 
in one or more of the actual tas^ks performed by the incumbent',' and the 
'theory" information must be referenced by' the student (or job incum- 
bent), rather than direct procedural instructions from a procedurally 
oriented job manual. But theory of operation may apply to 'vari ous kinds 
of content; what the system is supposed to do and how i-t does it in • 
principle, criteria of system effectiveness, environments in which the 
system is to operate, and interdependence of subsystems, physically and 
functionally. Theory of operation may include some, and"' possibly all, 
of the nomenclature associated with learning and performing tasks by 
incumbents. 

The instructional 'vehicle requirements are essentially those for pre- 
senting text, diagrams, and pictures (static and dynamic), and for 
receiving and evaluating symbolic responses from the student. 

Task Nomenclature (Identifications, Locations, and Names) and Facts, 
Rules 

This stage may be merged into, or be an extension of, the orientation • 
and. familiarization phase. The student learns to identify and differ- 
entiate task-related entities (situations and objects and operations) 
and to associate names with them. By learning nomenclatures, the 
student can be briefed and symbolically guided by the instructional 
vehicle. This enables symbolic rehearsal of task actions. It also 
facilitates the student's ability to mediate between situation and 
9ve''t task response, by giving himself instructions wi th words and 
images when the nomenclature is organized into descriptions of task 
procedures. 

The instructional vehicle requires the ability to present -the appearance 
or image of the entity paired with its label. This pairing of appear- " 
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ance and name may be presented by a variety, of means. In general, 
identifications of things are learned more quickly when they are 
shown in meaningful contexts. 

Facts, such as the color coding of icesistors for their resistance values 
or the weight and dimensions of a, gtven. piece of apparatus, can be 
learned with similar vehicle 'properties. 

The learning of direct symbol translation rules, such as for convert- 
ing inches into meters, depends on minor if any change in properties 
to the instructional vehicle for facts and nomenclatures. The proviso 
IS that the rule is a simple, one-for-one, translation of elements in 
one symbol set (such as degrees in Fahrenheit) to elements in another 
symbol set (degrees in Centigrade). 

Task Formats at the Conceptual Level 

At the termination of this phase, the student is able to give a more 
or less specific description of-,what initiates the task or duty being - 
studied, and of the major sequences of activity that comprise the task. 
Where operational task formats are learned, the student should also be 
able to state or identify the major variables comprising (a) the task 
situation or problem, and (b) the goodness criteria in the task outcome. 
In another form of test, the student is presented wi th motion picture 
sequences of the task being performed and he describes what is going 
on and why. 

if 

This phase may be more specific with respect to job tasks than famil- 
iarization, but less specific than the learning or memorizing of the 
descriptions of step-by-step procedures. The Navy officer does not 
usually learn how to perform ^the actual task itself, but gets a ' 
•working knowledge" about the^ performance of the task. 

A task format at the conceptual level is a general map of what is done 
•in disassembl ing-replacing-assembl ing an electrical or mechanical 
assembly, the general pattern of oper'ations in sighting a piece (or 
class) of fire control equipment, or the general steps in making a 
chemical analysis of the water in a boiler. This conceptual knowledge 
may become a cognitive "organizer" for actually learning the task as a 
verbalizable prpcedure, and actually performing it. Because the concept 
ot task format is not formalized in Navy training documentation, this 
IS an implicit category" at present. Its meaning as a training ob- 
jective can be interpreted in the content of some theory courses. 

The- training vehicle presents symbolic materials and the student makes 
symbolic responses. In addition, the presentation may include live 
demonstrations of task performance, or representations of it, ranging 
from text descriptions through motion' pictures . More sophisticated 
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and "enriched" presentations may present the task action as a "cause" 
and some system outcome as a related "effect". This material 



ful 



as a training experience to the extent that it is remembered and 



transferred to rel evant work si tuati ons , or to later traininq 
situations. ^ 



Note that this stage of learning is equivalent to the content of instruc 
tion to be specified in Subtask 05.22, Identify Type of Task Format 
If ana ysis was made in Subtask 05.22, its results can be carried over 
flit nl^ol^l° ^^'^ topic and possibly the next topic below. See Sub- 
task 05.22 for a more analytic description of task formats. 

Procedures Learned at the Verbal Level Only 

JescHbe'Jn tT.U^?.^'\^'' completion of this phase when he can either 
describe in detai the steps in the task procedures, including the signs 
or cues that specify the task response to make, or preferably when he 

vPrb.i^v''^' S'k''"'' °^ '^' '''^ and environments es?? Se 

verbally, and by pointing, each step in the task procedure. 

It is possible to memorize the verbal equivalent of a set of printed 
job instructions The evaluation of incumbents on their profic ency in- 
the training path system used in submarine and fire control contexts 
puts heavy emphasis on the ability of the incumbent to give verbal 
descriptions of procedures to be followed in the array of tasks makinq 
up a requirements pattern. The verbalized procedure may serve as a more 
effective precursor to learning the live context procedure if the 

tfnnf.nH'Ihr '^^^ I'"'?''"' ^^e _ ae£earance_ of ^the presented task situa- 
tions and the overt task operations as the incumbent performs them This 
implies effective learning of "nomenclature - identification and loca- 
tions and names" and their presentation in the context of re earsig or 
practicing the steps in the verbal procedure. rt^ntdrsing or 

Devices more sophisticated than printed text can increas'e the efficiency 
of learning the verbalized procedure, its reliability, and^ecial v 
Its transferabil ty to the actual or simulated physical Irk sitlat ^p 
acceotance farn.Hp Pr^°^^;,^^- But the dismay and student response 
?a^h\^'"th1n;^S;I^^I^.°p^e^1n^:^]^ns'^"^^ .ay be symbolic representations 

Performing Task Components with Guidance 

This level corresponds to actually carrying out 'the, task in full, but ' 
with some degree of external prompting. 

In Navy language, a skill consists of a "doing operation" - that is 
. P^^f°^"^i"g a task with guidance is equivalent to 

a middle stage of learning" - where the student is givinq himse f verh.l 
instructions in task performance (mediated performance wthqrownq 
degrees of independence from outside sources of task information ^ 
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: Doing the Entire Task/Job Procedurally: Barely Acceptable Mastery. 

I^HpH^'Kn!^ or incumbent is able to make his way through the task un- 
nlttiL 5 with errors and fumbling and at a slow, uneven pace; he is 
?h s shotllfhT h'° tasks or ongoing activities at the s me t me. 

cMhLt ^ equivalent of a late stage in "middle learning", where 
subtasks are getting organized into performance, but where getting 

caDa°c1tv Th^'°^'n"r%'?''''*J^ "^'^ '""^t °f the student's attention 
Tnnf,nl l' °t,his experiences has not yet been great 

aenrio. nn^'h h '."'/^ repertory of response patterns for contin- 
behaiin^.rpipmLj! practice in doing the task to make its 

behavioral elements more or less automatic, like subroutines This 
" erfnl"." equivalent in the Navy evaluation ex ressi of- ' 

performs task procedures without supervision", if the expression 
fomance"^^ ' °' """"^'"'^ acceptable standards for job per 

This level may be reached after sqine hours of procedural practice"on 
dJ^iJpH "JS^Pr*' V^^ ^"^P °^ "^^""^^^ of instruction^ and/or the 
detlHn..% T ]"^tructor as guide. The performance evel 

■ deter orates rapid y during a few days of nonpractice and is not 
rapidly restored with renewed practice. 

Highly Proficient Task/ Job Performance in Work Context - 

The student or job incumbent performs the entire task without hesita- ' 
oZI]L"J''T: that^indicates that he is anticipating and mak g 
provision for future steps as he proceeds; he is also able to attend 
to and respond to other concurrent activities. 

This is a learning phase and a skill level not identified in most course 
objectives, nor in incumbent evaluations. As a stage n learMnq H it 
Lsk'o.er'aHnn'^^ ' high degree of. "automatized" performance o? Jout nlzed 
task operations, including components of many decision-making ooerations 
Which now are performed with comparatively little mental "figuring out 
tas\1 " ro ;anv'.'h'^' level the student or incumbent can perfo m a 
;!fhpE^u^^"^^^i.h^ optimize resources across a number of stecs 
rather than proceeding merely one step at a time. Not only can he 
plan severa steps ahead, he can" anticipate conditions and suSations 
perform efficiently as well as effectively, and keep in mind soecal 

sSe'ci-??c?;ha't°he l'r^nT?T'l'J'' priSn-L" 'an8' h' . 

hv^w! nn I could not do If his attention was completely dominated 

by trying to remember what next to do. This level imolies that manv 
procedura operations (whether correct or incorrect have been learned 

e?fer J''' J''" automatic, and -that^fs ^ntic p t o s are 

effective in transitioning his activity, smoothly, from one sequence of 

per oTo''?ir''r: This level of skill is wel i-re?ained ver' ong 
periods 0. time and is quickly relearned from any degradation in recall 
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l^.l^J^^^V^ reached only after extensive and varied repetitive 
practice that makes the full demands on attention required n^he job 
But It can be erroneous to assume that only job experiences in area? 
number and variety can lead to this stage. We shK recoqn L^Sat 
job experience is also a learning experience, and any iSJq expe- 
nence is potentially subject to deliberatelJ induced train na Trtt.. 
Wrong procedural responses can be overlearned as readily as c^^^^^^^ 
procedural responses. It is a good working hypothesi tha? manTiob 
tolHr%^^° could reach the level that combines smooth re ^b e 
b m e ?aif°;f dn\'J °f response and strategi'c ! 

bv trainin?iL H reasons that could have been controlled 

Dy training type decisions and actions, and that many others arp Hp 
layed from reaching this level for the same reason 

ran'Sr of^L^k'^!!!^^"^ requirements, both in kind and 

oT?e%o°n^se"l^a?;a\1o?a^V?:^^^^ 

ance functions. Plus very extensive blocks of tiL for pracl?ce^ 
Unusual Task Conditions 

This is learning to cope with the multitude of thinqs that ran nn u,^nnn 
earning burden of exceptional conditions to c^pe juh Unlfss Z 
learneS fn 'l/l'^f'^^'''^^ redundancies in the coStent o? what is 

Berforming Job/Tasks at Key Man Level 

This is demonstrated by performing the job at a level nf <;kin fh.f • 
recognized by others as unusual ly proficient J iLrfh! ^- ' 

a'Sinof '° '°'i'''''t 4ef^iIe^•n^^• a o^Vw e ^'fh^ s'^ 
having the same job are by-passed or have failed. 
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There can be a combination of training, job experience/aptitude and 
motivations that make some individuals the key technical men in their 
organization and specialty. These individuals tend to contribute far 
more than their share to the mission effectiveness levels of their 
organization The key person phenomenon is likely to be more than a 
mere sum of skilled performances, hence applicable to the person rather 
than merely his skill attributes. In some cases, the key person may 
not have done well in academic phases of training such as often 
demanded by the conceptual demands of theory of operation or verbal 
memorization of procedures and other rote demands. But the p ractica l 
importance of such individuals in organizations makes it desirable 
to Identify and characterize both the role and the skill level in any • 
complete range of description of competence. 

The development of the key man probably requires substantial catering 
to individual differences and more or less unique training routes. 

Refresher Learning 

At the beginning of refresher training the student cannot perform or 
recall to criterion performance levels, tasks a'nd task information 
which he was once able to perform, but at the conclusion of refresher 
better"^ ^^^^ '° perform at his once established level or 

This is a "stage" of learning intended to compensate for the forgetting 
effects of extended periods of time with no practice of the skill and/ 
or intervening experiences some of which may interfere with the recall 
or the correct responses to make in the old task. 

If the tasks were learned to high levels and performed a large number 
Of times, recall with practice may be rapid. But the rapidity of re- 
call and full reinstatement of the original skill, may be slowed or in- 
creased depending on conditions of refresher practice. The princiDle 
^L'lnt'^i^' retraining of an old skill is that of inducing any other 
content of retention: the greater the extent to which the original 
situation and all of its psychological context is reproduced, the 
???LtL^^' probability of recall of the responses made to the original 
^n!J^^l ^ef^esher training, a highly significant factor in that 

context IS the actual making of the responses (cognitive and motor) under 

nS? .'??n ';!;^'^'°"' °^ "^^^^ ^'^ t^'^ °^^"gi'^l learning Th s is 

not altogether a paradox. 

"Iu^%?;t^^^''P^^-, memorized a poem. Years later you are given 

the title and asked to speak the poem. Not a single word can be re- 

nhiLo" 1 ^P'^ll^^ "^^Tl.V"}^ °P^"i"9 Ph>"ase. You speak the opening 
phrase aloud. The probability of your being able to continue the words 
of the poem vastly increases. As you complete the utterance of the 
first line of the poem, the probability of your recall of the second 
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line increases still more. If all sectioris of the poem were learned 
equally well, each additional phrase you speak continues to increase 
the probability of remembering the remainder of the poem. If you 
learned it by speaking it aloud, it will best be recalled by speaking 
It aloud. The recall is assisted by the. "response context" which, 
especially in serial activities, becomes the most significant set of 
cues for recall. But the effectiveness of response context is not 
limited to serial activities. 

This principle helps to understand that attempting to relearn the 
skill by practice which simulates the original early learning conditions 
will be highly inefficient for redintegrating the old skill at its ' 
terminal level when it ceased to'be practiced. The Jcinds of responses 
made in early practice are not the kinds of responses made at full 
competence level. Relearning an old; skill should attempt to simulate 
the terminal levels of performance of the skill. When its psychological 
context has been restored, practice may concentrate on ironing out the 
deficiencies. 



Studies have shown , that procedural skills, such as emergency operations 
in an aircraft, if once learned to a higi^ level of performance reli- 
ability, can be "refreshed" as effectively by verbal rehearsal as by 
practice in the simulator of the actual procedures. No doubt the verbal 
rehearsal, to be effective, must be accompanied by some internal imagina- 
tion of the cues and responses connoted by the verbalizations. It is 
unlikely that procedural skills learned only -to intermediate levels can 
be effectively refreshed by symbolic rehearsal. Skills dependent on 
complex perceptual -motor or motor patterns will probably have little or 
no refresher value derived from verbal reh'earsal , except perhaps to 
alert the operator to a changed context of cues. Example, the American 
driver who has returned from driving in the United Kingdom, when getting 
into his car at the airport reminding himself that in the U.S.A. one 
drives on the right rather than the left of the road, and rehearses some- 
mental imagery to that effect. 

Add this Stage of Learning characterization to the description of job task 
skills that emerged from the determination of relevant job tasks in the 
Navy inventory of jobs, Subtask 05.14. The assessor can now formulate a 
search argument for retrieval of relevent courses from the course descrio- 
tion data base. . 



COMMENT 



The foregoing definitions of learning and performance levels are intended to \ 
help assign the relevance of an innovation in- training by persons with some 
background in the psychology of learning and training. Existing Navy terminology 
and distinctions were adapted where possible in this determination of relevance 
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Li Sn f ncepts of learning level described here are represented in formal 
Navy documents about course objectives on one hand, or perfomance crileria on 
the other. The possibility should exist that Navy practices Sav be ailp^cf 
moderately changed to adapt to the disl^fnctions described here anS fV^.H 

llllZ X T''l' '° ^r?^^^^^' innovations ?n tSain e n lo ^ or ' 
SSl least overall cost-effective level of learning fol?6wed bv a ■ • 

■ e?y' J\"rth?n.:;'!''A"~''^~^°' ^^'^"^^"^^ and\'?:n"ffeJ°|]t s ?e , ' 
formll t^afning. ^"''^ °^ bare procedural mastery" which terminates much 
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TASK: DETERMINE RANGE-OF-EFFECT 
SUBTASK: IDENTIFY RELEVANT COURSES 



DESCRIPTION 

At thi 
tified 



JL^i^ ^" ^^^^ content skill route, the following have been iden- 




\ ■ ■ 

The. general class of skill 

The task structure elements that are applicable 
\The relevant job tasks from the Navy inventory 

Thfe applicable stage(s) of learning, including formats for procedures, 
deciv^lons, and constructions. 

This infonr^ation should enable a variety of search strategies to be applied 
to >a data ba^se of course and course objectives descriptions. 

As in search operations described previously, the two stage activity consists 
of us^ing the search specification to get a list of candidate courses poten- 
tially relevent to the innovation, and then validating -or rejecting each 
candidate on the basis of informed judgment based on at least a general know- 
ledge of the course content. Mere descriptive labels of courses and course 
objectives will generally be insufficient for the validation' by-judgment phase. 

Because many courses are set;jp to fill standard time slots in t^rms of weeks, 
there is bound to be arbitrariness as to the stage or level of Jskill that a 
•given course will carry a body of students. This fact justifies a "stage of 
Skill Learning" characterization applied to the innovation. 

The identification of courses in the curriculum relevant to the innovation 
may also have to depend on the names and objectives of training units within 
the courses. Unfortunately, if there is not a match between the nomenclature 
used in the search specification and nomenclature used to describe the train- 
ing units and their objectives, a knowledgeable person may have to mediate 
the translation, , . ' • . 

The identification of training units within courses will have to be made in 
the naxt step, if it has not been done here. 

PROCEDURE 

1. Identify by title and Vther identifying code, each course of instruction 
current or projected, that fits the criteria of relevance to IhrinnoJation. 
nJ.'^Sifmonfc ?° ^de"J]fied will presumably contain one or more training units 
tion. (o^ portions of the course) specifically relevant to the innova- 
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^* a'deteIlm?naiZ'n^';!;^i"?''^^ on-the-job benefits should .have 

, mafbTes;ab]?sheJ.''' T'" '° '''' ' '''''''' ''''''''' 



3. "The relevant training courses shoald be unambiguously identified so that 

can find the correct reference 
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TASK: DETERMINE RANGE-OF.-EFFECT . ' ' 

SUBTASK: IDENTIFY RELEVANT TRAINING UNITS WITHIN COURSES 

■ •■_ • . 

DESCRIPTION 

Much or all of this subtask may have been, accompl ished in the process of 
identification of relevant courses, Subtask 05.16. The determinatioJ of 
training unit applicability may have been somewhat perfunctory. Furthermore, 
the identification of merely one relevant training unit within the course would 
justify including the course as relevant, without examination of the possibility 
of additional training units also being applicable. . 

The present subtask, therefore, requires an examination of all of the constit- 
uent trainiiig units and objectives with a course of instruction for meeting 
the contextual, as well as descriptor, matching of innovation to training re- 
quirement and training process. All previous analytic descriptor selection 
and search operations have focussed forward to the decisions of acceptance or' 
rejection for relevance to be made' in this subtask. That is to say, only 
candidates for acceptance have been selected. 

PROCEDURE ' ' 

1. Review carefully the description of thp innovation in Task 1. Note not only 
its tar:geted objectives , but the contexts in which the empirical data were 
developed, and other substantive aspects- of the innovation. This context of 
information should be held in mind while examining the context of the 
descriptions of each candidate training unit within 'the course. 

2. Examine the context of the "potentially, applicable training unit with equal 
care. A guide for comparison is not so much in the training operation-as 
in the clear identification of the specific skill or skill component that" 
is to be learned by the student. This is an operational reference which 
IS generally better than merely 'a verbal reference. The judgment of rele- 
vance may be facilitated by anticipating -the next subtask by asking, "Will 
the innovation really improve, or not improve and impede the skill element- 
that is being learned here?" If a strong doubt exists,- it may be better to 
reject than to accept a candidate, so as to avoid unduly inflating -the 
estimate of potential benefit from the innovation! 

3.. If the innovation aims at skill components that 'are clearly not identified 
in otherwise clearly relevant- courses, these components must be so identi- 
fied so that they can be' evaluated as on-the-job benefits. The reference 
course should be identified, and the training units or objectives most 
appropriate by the acquisition of the added skill identified. This will 
enable a cost analysis for adding the necessary training to be balanced 
agains-t assessment of on-the-job value. 
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TASK: DETERMINE RANGE-OF-EFFECT 



SUBTASK: ASSESS THE TRAININ^ BENEFIT PATTERN " " 



DESCRIPTION ■ ■ ' ' '■ ' . , 

/ . K 

'The assessment procedure on 'the task- content route has^td this poTnt, .identi- 
fied the courses and training units ..within courses that. are relevant .to the" 
innovation, in this .step, the magnitude of the benefit will b'e assessed' the 

J^fft?'"'/ ^'lt 'i;^ magnitude Of the .benefit weighlei. and thl proba 

bility of achieving the benefTt estimated. . ' . f"'"^" • 

" ■ - . • ' 

The major target types of benefit consist of: training time reduced, aptitude 
level reduced, attrition level reduced, and performance level increased or 
extended. The benefit targets that can be most readily translated into cost ° 
. advantages are reduction in traini/g time and reductions in attrition levels. . 

•Note that when relevant courses have been identified, the cost assessment model, 
Vl\ the number of students and other multiples of the unit of benefit 

that have been estimated for the course. • u' ueneTiL 

PROCEDURE 

, 1. Answer ques^tions about reduction in training time. 

' 'Each of the following questions should be applied to each of the trainina 
course elements that is potentially relevant to^ the immation I? mSre^ 
than one judge makes independent estimates about the same entity the 
eyalua-tion^program will average the estimates, but also use the distribu- 
«, tion of judgments as a measure of reliability of ji 



uun OT juagments as a measure of reliability of .judgment about the effects 
of the innovation on that training course element erreuts 

2V^stion lA. For 'each training /element (or unit) relevant to the in nova- 
t on, what IS -the estimated p ercentage reductionSn training time fn" th. t 



The estimate will be a judgmental extrapolation from the description of 
the innovation made in Task 1, and any empirical data contained in that 
description, modified by subsequent investigation by the assessor This 
estimate assumes that the innovation is technically feasible and s applied 
in moderately favorable conditions. appneg 

If.however, the assessor believes that if the innovation is introduced 
training time will actually increase rather than decrease or remain as it 
IS, he should go to Question lA Negative. 
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Question lA Negative 

Assume the assessor believes the innovation wil.K increase traininq time 
over comparable conditions. He would specify NEGATIVE and the percentage 

r!;"^;^ ^'"^^ increased over the standard or projected training 
time for the training element: H\would then answer Questions IB, IC and 
ID as they are put,, but with referee to the negative value.' 

TtldllT. Itf !°l iT^ °I the next fiVvears. what is the nun-i;er of ' " 
, students that will be a frected in that traininq element, ov- unit? ~ 

Question IC. What is the relative importance ^of this pa rticular benefit 
■ Innovation" '' °^ ^ '"'^'^^ (°r 11 abfh ties) offered bv ^ 

Use the following scale for estimating importance: " 

. 1 2 3 4 5 6 7 8 9 '10 ' - ' 

' . trivial I moderate I crucial 

importance I . . I • 

identifiable very 
■ " ' but minor substantial 

IIlinJHnTJj benefits that may be derived from an ^ 

! ?hL^I?S- ? t)^weighted against each other. It also enables a liability, 
aaa nSf^ as part of the evaluation, to be weighted ^ 

against other labilities^ and against benefits. The assessor may thus 
express the value he assigns to qual i tative influences that should empha- 
size or de-emphasize quantitative expres-si-Sns of benefits and liabilities 
Jr.in?Kr''H" with each other, with a general reference to the Navy's 
training and operational mission. . ^ 

of^thI°benefitf '^ ^^V*^"^^^^'^ ^' °^ achi eving the estimsited iftaqnitude 

I=f?!l^^^^'°" assessor identified the magnitude of the benefit esti- " 

Sp. th°.°f'n ""der moderately favorable conditions. The assessor now 
makes the following two estimates. 

a. Wha\t is the estimated magnitude of the benefit under conditions that 
wo^\d e^ favorable? This value should exceed, b^ some amount ' 
conditi-oUs magnitude of the benefi t^iHi^ moderately favorable 

^' S^J wnMfIl\"J-''M^ °I magnitude of the benefit under conditions 
that would-be highly adverse to it? This value should be less than 

' cSndft.nnf ' rl ^^^^^"^^anitude of the benefit under moderat^l^favorable 
conditions. The value could become a negative one, in which case the 
proposed innovation would be a liability with regard to this Variable 
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Answer questions about reductions in attrition rate during and at the end 
.of training. 

The innovation may be aimed only , at a reduction- in training time, holding^ 
attrition rate constant. Or the innovation may be, aimed only at a reduc- 
tion in attrition rate, while holding training time constant. The third 
alternative may attempt to combine reductions both in training time and 
..attntiop. The topic here applies to the second and third alternatives. 

Estimation , on chan^'e in attrition rate should focus on each training 
element or unit found relevant to the innovation. The standard formula 
used to express attrition rate in Navy training will define the present- 
meaning'of attrition. 'One should also assume that present standards or^ 
criteria of acceptable performance would apply to the innovation. 

Question 2A. For each training element (or unit) applicable to the inn ova- 
tion, what IS the -estimated percentage reduction in existing attritio n rate 
(or normaTTy projected attrition rate) to be expected from a pplication of 
the innovation? ■ 

Because attrition rates are expressed in percentage figures, the estimate 
will be a percentage of a percentage. Thus, if the normal attrition rate 
for course X is 20%, a reduction of the attrition rate by 30% would reduce 
the existing rate by approximately a third, to 14%. But because proportions 
.s_eem generally easier to estimate than actual values, estimate the percent- 
age (or proportiorp^^of reduction of trainees normally attritioned that would 
not be attritioned if the. innova-tion were introduced. Thus, estimate the 
percentage reduction in the existing attrition rates if this innovation 
were to be applied under moderately favorable conditions. 

If, however, the assessor believes that under conditions moderately favorable 
to the innovation, the attrition rate will increase rather than decrease or- 
remain the same, he should go to question 2A Negative. 

Question 2A Negative 

Assume that the assessor believes that the innovation will worsen the other- 
wise expected attrition rate. He would specify NEGATIVE and the percentage 
MLn^^ur^^^I!^^ students that would be increased.- Thus, the expression 
NEGATIVE 25% would mean that a quarter more than the present attritioned 
students would be attriticned if the innovation were introduced. 

He would answer Questions 28, 2C, and 2D as thty are put, but with reference 
to the negative value. \ 

Question 28. What will be the number of student s in the training element 
or unit in each of the next five years? 
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Question 2C. What is the relative importance of this particular benefit 
in relation to all of the other benefits (or liabilities) offered by the 
innovation? 

Use the following scale for estimating importance: 
1 ! 2 3 4 5 6 7 8 9 10 



trivial j 




crucial 


importance 1 


moderate j 





i identifiable very 

but minor substantial 

This judgment enables the various benefits that may be derived from an 
innovation to be weighted against each other. It also enables a liability, 
if identified and assessed as part of the evaluation, to be weighted 
against other liabilities, and against benefits. The assessor may thus 
express the value he assigns to qualitative influences that should empha- 
size or de-emphasize quantitative expressions of benefits and liabilities 
in comparison with each other, with a general reference to the Navy's 
training and operational missions. 

Question 2D. What is the probability of achieving the estimated benefit 
in the reduction in attrition? 

In Question 2A, the assessor identified the magnitude of the benefit esti- 
mated to occur under moderately favorable conditions. The assessor now 
makes the following two estimates. 

a. what is the estimated magnitude of the benefit under conditions that 
would be most favorable? This value should exceed , by some amount, 

the estimate of the magnitude of the benefit under moderately favorable 
conditions. * - 

b. What is the estimate of the magnitude of the benefit under conditions 
that would be highly adverse to it? This value should be less than 

the estimate of the magnitude of the benefit under moderately favorable 
conditions. The value could even shift to the negative side and be- 
come an increase in attrition rate as' compared wi th present attrition 
rate. * 

3. Answer questions about reductions in aptitude requirements. 

"V 

Assume that the intent of the innovation is to reduce the normal or expected 
aptitude requirements for achieving course objectives, while maintaining 
existing training time and attrition rate as constants. 

In this- context, an "aptitude" is defined as a score on standard selection 
tests that are used for the classification of Navy personnel, which is a 
minimum score for acceptance to a given training program. By lowering a 
minimum aptitude level , more persons become potential candidates for the 
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course or training program. If the cutoff aptitude scores are on the high 
slope of the distribution of the ability, small reductions in aptitude 
requirements may result in very large increases in potential candidates. 

A reduction in aptitude requirements for a given training objective will 
be valuable if the Navy sees: (a) shifts in overall demand for higher 
level aptitudes, thus increasing competition for higher aptitude people; 
and/or a snift in supply of recruits towards lower aptitude levels 
Either of these issues presents problems that are not subject to resolution 
solely in cost justifications. They will, therefore, not be directly 
addressed by the present procedures, although the issues may be expressed 
qualitatively in the following format. 

It is possible within a given training context to assess the value of the 
innovation applied to aptitude level in terms of training time and attri- 
tion rate. Assume that the innovation claims to be able to reduce the 
relevant aptitude scores by "5 points". In existing training procedures 
(without the innovation), the people within these 5 points on the apti- 
tude measures would require an average of 20% more training time, as 
compared with the average training time of those students within present 
n? .no/ levels. In addition, the new group would have an attrition rate 
of 30^ whereas the attrition rate of the old group was 10%. These are 
cost factors which can be used to evaluate the innovation. 

Assume that the range-of-effect analysis has identified the training 
.'nH?^H/^^^''"* the innovation. It is also assumed that the minimum 
aptitude scores relevant to the innovation have been estimated if thi<i 
estimate has not been made, this estimate should be a part of question 3A. 

For each of the relevant courses, the following questions should bfe 
answered in making both quantitative and qualitative assessments. 



Question 3A1. _ What is the proport ion of students in the innovated class 

w ho would be introduced into the cTasTTf the in novation wprP t.n hp 

appl ied? ~ ■ ^ 



This would be an estimate of the proportion of the average class size that 
would fall below the minimum existing aptitude scores and above the new 
minimum aptitude score. For example, the range of the reduced aptitude 
levels could account for 20% of all the students in the revised course 
makeup _In future questions, this group will be called the "affected 
students . Assume the innovation has been introduced under conditions 
moderately favorable to it when making this estimate. 

Question 3A2. What would be the average additional trainin g time for the 
affected students to reac h the criteria of the course ob.iecti ve5;~7^?rTH^ 

assumption that present conditions held, and the innovation i;j??ninT 

introduced? - 
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It is possible to use existing experience in the course which relates 
level of aptitude to class performance for extrapolations to the lower 
aptitude levels. 

Question- 3A3. Assuming the additional training time estimated for the 
affected students without the, innovation, what would be the estimated" 
attrition rate for the affected studentsT ~ 

It is recognized that it is very difficult to make judgments about two 
interacting variables. Nevertheless, there will be a proportion of 
students who would have to recycle, even with an extended training time 
for the average of the affected group. Furthermore, there would be 
inevitable washouts who couldn't learn the material even with very 
extensive training. 

Question 3B, For each of the next five years, what is the number of 
students in the class of course of instruction affected by the innovation? 

Question SC. What is the relative importance of this particular benefit 
in relation to all of the other benefits (or liabilities) offe red by the 
innovatiolT? 

Use the following scale for estimating importance: 

1 2 3 4 5 6 7 8 9 10 



trivial | 




crucial 


importance I 


moderate j 





identifiable very 
but minor substantial 

This judgment enables the various benefits that may be derived from an in- 
novation to be weighted against each other. It also enables a liability, 
if identified and assessed as part of the evaluation, to be weighted 
against other liabilities, and against benefits. The assessor may thus 
e)(press the value he assigns to qualitative influences that should empha- 
size or de-emphasize quantitative expressions of benefits and liabilities 
in comparison with each other, with a general reference to the Navy's 
training and operational mission. 

Question 3D. What is the probability of achi eving the estimated magnitude 
of the benefit? ' — '■ ^ 

In Question 3A, the assessor identified the magnitude of the benefit esti- 
mated to occur under moderately favorable conditions. The assessor now 
makes the following two estimates. 

a. What is the estimated magnitude of the benefit under conditions that 
would be most favorable? This value should exceed , by some amount. 
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the estimate of the magnitude of the benefit undey^ moderately 
favorable conditions. 

b. What is the estimate of the magnitude of the benefit under conditions 
that would be highly adverse to it? This value should he less than 
the estimate of the magnitude of the benefit under moderately favor- 
able conditions. The value could become a negative one,. in which ' 
case the proposed innovation would be a liability with regard to 
this variable. 

Answer questions about supplementary statements of benefits and liabilities. 
This would be Question section E to be added to Question 1 or 2 or 3, 
depending on which was answered by the assessor as the primary target of 
the innovation. This section is also a proper supplement to benefit evalua- 
tions derived from the Instructional Vehicle assessment route (Subtasks 
05.02 to 05. '08) and to benefit patterns applied to on-the-job benefits 
assessments. 

Study of the innovation in the context of the evaluation process may bring 
to light a number of liabilities and benefits possibly overlooked in the 
assessment format. The assessor is invited tD identify such additional 
benefits and liabilities, state them qualitatively and, where possible or 
applicable, quantitatively. By liability is meant an undesirable outcome 
of the application of the Innovation NOT an impediment to the realiza- 
tion of the innovation. The assessor may. see long-range effects, or side 
effects, or backlash effects not directly evident in the innovation. 

The assessor should also review the data obtained in Task 3, Make Prelimi- 
nary Feasibility Profile, and reaffirm or modify the feasibility factors 
as evaluated at that time. These factors may suggest additional benefits i 
and liabilities inherent in the innovation. 

As in his other estimates and predictions, the assessor should express, as 
best he can for each statement of benefit or liability on a variable, an 
estimate of its magnitude, its probability of happening, and its relative 
importance. An outline of format is offered in Figure 5.18.1. Where 
possible, the variable or factor which names the benefit or liability 
should be expressed as a qunatifiable entity, and an. estimate of the 
magnitude of unit benefit or unit liability be expressed. The size of 
the multiplier of this unit benefit should be determined or estimated so 
that the full scope of the benefit can be expressed in the same way. that 
the preceding- variables (course length, attrition rate, and aptitude level) 
have been expressed. This would enable formal analysis to be made. 

But even where quantitative magnitudes and the numerical size of the 
multiplier factor are not feasible, the identification of the supplemental 
benefit or liability should be made and accompanied by estimates of proba- 
bility of occurrence and relative importance. Such statements may not be 
processed formally through cost models, but presented to the decision maker 
in the form on which they are set down by the assessor. 
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n^^S'Ihn?f^•^M ^ structure for describing the supplemental benefit 
or liability. It is probably not desirable to make a printed form for 
Aril assessor may be constrained from getting and setting 

he ?s ?ShlMS!i I"^?"^"^?*^?" essential to understanding the message because^ 
he is inhibited by lack of space or lack of suitable category heading. 

COMMENT: The data developed in response to the previous procedural questions 
shou.o De prepared and formatted according to the demands oblate? task to 
aSaVsfs?^ cost-benefits analysis and economic ' 

^" Analysis "^^^^ 'developed in this subyask to Task 6, Perform Cost-Benefits 
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1. Identify the name of the variable or factor which is a supplemental bene- 
fit or a liability. 

2. Estimate the magnitude of effect of the benefit/liability. 

a. If possible, quantify the unit of magnitude, and the multiplier 
factor, to apply to the unit of magnitude appropriate to the 
intended scope of the application of the innovation. 

b. If numerical quantification is impractical, express magnitudes 
in relative terms. 

3. What is the number of entities or instances (jobs, courses, students, etc.) 
to which this magnitude of ben.efit (or liability) applies? 

4. Estimate the relative importance of the added benefit (or liability) to 
the overall effectiveness of " the innovation. 

• Use a scale from 1. tb 10. 



3' 



10' 



trivial 

importance 



iden 



moderate 



ifiable 



crucia 



but minor 



very - 
substanti al 



Any additional statements that may add or qualify information for the 
decision mJiker may be added infonnally. 

. Estimate the range across which the magnitude of benefit (or liability) 
may occur. 

a. The estimate of magnitude made in item 2 presumably assumed ' 
conditions moderately favgrable to the benefit occurring (or 
moderately unfavorable for the liability to occur) at that 
magnitude. 

b. Estimate the magnitude of the benefit under conditions highly 
favorable for the innovation.- (The converse for a liability.) 

c. Estima'te the magnitude of the benefit under conditions highly 
adverse to^the innovation. 

6. State reasons and justifications for these expectations. ' \ 



FIGURE 5.18.1 FORMAT FOR DESCRIBING A SUPPLEMENTARY 
BENEFIT OR LIABILITY 
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TASK: DETERMINE RANGE^OF-EFFECT 
SUBTASK: SELECT TYPE OF KNOWLEDGE 



DESCRIPTION 

A key factor in the economics of training is that, unlike most skills, most 
knowledge can be learned by demanding only symbolic responses from the student. 
The presentation may range from text, diagrams, photographs, and motion pic- 
tures, to computer generated displays.. But the students write or vocalize 
answers or push answer buttons. Knowledge is symbolic information about 
things and actions, and rules, whereas a skill is manifest in actionFlper- 
formed.as tasks with purposes and goals. Knowledge may support skill, but is 
not equivalent to skill. j yy' . , uuu 

Unlike skill training, which is more or less directly tested by its applica- 
bility to job demands, knowledges can be 'tested for their contribution to 
learning and on-the-job performance only by elaborate empirical tests which 
are rarely made, and usually have discouraging results unless one is concerned 
with where to cut costs. But because knowledge training. is relatively inexpen- 
sive to administer, as compared with skill training, it widely persists.- 

Distinctions should be made between reference knowledge and enabling knowledge 
A reference knowledge is information context that is not directed to a task or" 
skill, but to various.kinds„of environments and ideas in which the worker works 
and the job is performed. Reference knowledge may have motivational value, such 
as when demonstrating the consequences of accidents that followed violation of 
safety procedures. 

An enabling knowledge is one which directly assists in the learning of a task 
skill or supports it in practice. Thus, the ability to verbalize a task pro- 
cedure, while not the task skill itself, assists in learning and performing it 
as a skill. Other enabling knowledges consist of factual data that are used 
m task performance. " ■ 

A major distinction can be made in the way that knowledge content is learned, 
retained, and recalled. A rote knowledge implies that verbal lists, tables 
or sentences are memorized and reinstated in a verbatim sense (multiplication 
table, stanzas of a poem). Materials learned by rote have a restricted range 
ot meaning, so that the content of rote material generalizes very little In 
general, materials of this kind are often learned by repetitive practice" How- 
ever, they may be better retained over periods of time when supported with 
mnemonic images; or when embedded in examples and other "meaningful" material. 

A second form of knowledge content is conceptual - it deals" with ideas rather 
than associations between symbols. Concepts are abstractions that apply to 
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classes of entities and" their properties. The meanings of terms like 
circuit , innovation", and "effecti vfeness" are the nature of a concept 
Concepts are subject to generalization by learning and by thinking" pro- ' 
rTnllnl ^"J^P^^ Organized into "principles" and higher order 

concepts. A_l earned concept has a label and a meaning or reference, 
although It IS possible to exercise the "meaning" of a concept without 
knowing or recalling its label. ^ wuiuui 

bll'jf '^"°^;'^39^ tested not by its mere existence in memory, but 

nLf! ''^h « a body of content when circumstances would make its Recall 

"f!^!:.. ? ^t 1^ the recall of the knowledge content that is 

knnSipH ''f iniP?':tance, but also the recall of the applicability of the 

knowledgeto a situation other than to a classroom t est question As an 

• the.^nowledge of human reaction times should -deter'the 
applied psychologist from tailgating drivers ahead oFFTuirat high speeds. 

It is possible that the following list of identifiers may assist in identifv- 
nnlfS. course item objectives to which an innoval on n reference 

knowledge training mightapply. Perhaps a better terminology wo Id be based 

? sJr r n^r?l '''^^"'^ *° ^^"^^ °f training objectives by Navy 

instructors If the names were used consistently. A key name which i/fairlv 
general is theory of operation" which, as identified' Jonsu^e iJy Jtas - 
room hours in instructional institutions everywhere. "-""^^^^ ^^^^ ^'^ss- 

Individual facts and organized clusters of knowledge, meaninqfullv connected 

mate'Sf's ?ll.°[n'''T?" ^n^o^niation which may be'essentiai to th Jo I 
mance of skills in verbal communication, as described in Subtask 05 13 To 
this learned knowledge should be added skills that make up the taskof 
oT5S7'"i;p? ^'^'f^ information". As indicated in the foff w s btask 
Sp \'f nir ?'' knowledge, often of more limited scope, can be support- 
i^ncI'JV"* ^^^^^^ essential - to many kinds of Navy tasks, includina 
those that are supervisory. ^-a^N^j, nn-maing 

The reader is reminded that an innovation that deals with cost factors asso- 
ciated with a vehicle of instruction - such as devices thaTpresent Pictures 
of things in motion -.may have its range-of-effect more efficiently deter- 
mined by the instructional Vehicle route, Subtasks 05.02 through 05^07 

PROCEDURE 

1. Determine whether the instructional content or process in the innovation 

vne'"' i'.'p^^h^° ^ ^^P^ °^ to an enabl S wl d e 

?uh?;.^ n^%^^'' determination on the basis, of the DESCRIPTION under 
bubtask 05.20, and the categories described in Subtasks 05.20 and'05.21. 



2. 



cIei'to^":b?:s'k°S5'2'g!"' exclusiveiyto reference knowledge types, pro- 
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^' TuJ^^, ''nr°o?*"'°" applies exclusively to an enabling knowledge, go to 
Subtask 05.21 . a a . a^^ 

4. If the -innovation appears to apply to both reference and enabling 
knowledges, follow the route beginning with Subtask 05.20 and Subtask 
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TASK: DETERMINE RANGE -OF- EFFECT 



SUBTASK: SELECT APPLICABLE TYPE OF REFERENCE KNOWLEDGE 



DESCRIPTION 

A reference knowledge consists of information about the tontexts of the per- 
sonal, technical and organizational situations in which the' incumbent and his 
work will be active. Thus, the information may be relate^ to the work and 
work environment, and perhaps the incumbent's living and social environment, 
but does not constitute a primary work skill, nor does it appear to have any 
direct bearing on the learning of the skill, except perhaps by motivational 
routes. An example of the latter might be satisfaction in being identified 
with important and prestigeful organizations and functions, or the opportunities 
for promotions and career development. 

Each of the following types of reference knowledge may have somewhat different 
instructional capabilities, but not necessarily. Obviously, all of these types 
could.be transmitted by classroom lecture or by textbook. But imaginative 
techniques could be devised for each type that could make it more effective — 
for whatever purposes training management may have -- as well as more efficient 
than the traditional alternatives. 



PROCEDURE 

1. Determine to which one or more of the knowledge types listed below the 
innovation is applicable. 

2. Even if the innovation's description points to a given type of course con- 
tent, or to a given named course, check its possibilities against the othe>* 
types and identify them. 

t System Purposes 

What'the entire organization and its parts are supposed to do and the 
various condi tions, under which its goals are to be accomplished. The 
concept of mission, and mission objectives, and of the values to be 
sought and maintained. The "system" may range from the Department of 
Defense, the Navy, or a branch of the Navy, to the ship or installa- 
tion and, more specifically, to the purposes and objectives of the 
arm and watch. The form of instruction may range from platform' speeches 
to motion picture scenarios to dynamic simulations. 

) ■ 

f Organizational Roles 

Thesecould be an extension of system purposes; what organizational 
^ entities have what responsibilities and authorities and spheres of 
influence. Typically, organizational roles are communicated with 
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organizational charts. Potentially, far more meaningful forms of 
presentation could demonstrate the cooperative and competitive 
functions of hierarchic and scalar relationships in organizational 
dynamics. • .. 



•, Contexts of Operation 




The interfaces between the incumbent's job and job responsibilities 
that impinge on his own. Also, the range of situations and circum- 
stances in which the incumbent is expected to perform his tasks in 
real environments. This might include the range within which his ' 
initiatives are to be encouraged, but outside of which constraints 
are imposed on his initiatives. y ^ 

• Theory of Operation 

This may range from theory of electronics to the construction of'and 
operating functions of the" classes of equipment the maintenance 
mechanic is to repair. In another type of job, dietary theory as 
potentially appl ied- to the planning and preparation of meals. 
Theory of operation generally applies to how something works. The 
content may be developed a't any of many levels .of discourse. It can 
serve as a basis for learning of nomenclature and identifications 
that later may be relevant tovskill learning. 

It is possible that the learning of a suitable kind and level or a 
theory of operation may enable some job incumbents to deduce task 
actions in the absence (or forgetting) of the applicable procedure. 
This assumption may underlie instruction in theory of operation to 
maintenance mechanics as a substitute for troubleshooting strategies 
and procedures. The practical justification for this assumption may 
be debated elsewhere. 

It is also possible for the learning of theory of operation to serve 
as memory aids for procedures, both in their learning and their reten 
tionand long-term recall. In order for this assumption to be 
justified, there would need to be a careful scheduling of instruc- 
tional sequences such that the two kinds of learned content would 
effectively support each other in the student's economics of learning 

Since theory of operation generally represents some kind of flow or 
sequence of process or action, a "dynamic display is suggested where 
process variables may be changed and the consequences observed in 
overall system behavior, or in the behavior of affected components. 

To the extent that learning theory of operation does facilitate the 
learning of a skill or enhance its performance, it falls into the 
class of enabling knowledge. 
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• Organizational Rules and. Constraints 



These may range from rules governing seccif^ity to the procedures for 
ordering a spare part. They represent realistic factors in both the 
working environment and the personal environment of the incumbent. 
Realistic instruction should probably include the kinds of -condi tions 
that justify the breaking of a policy and the means of making the 
justification. 

• Nonwork Directed Content of Instruction - 

A miscellaneous category. 'May range from personal hygiene and 
recreational facilities to interpersonal psychology. 

Specify the capability of the innovation to support one or more of these 
types of reference knowledge, and the key function or capabilities which 
enable the training or learning enhancement to occur. The^ description 
should be structured so as to assist in identifying courses and course 
content in which the innovation, applied' to these or other reference 
knowledges, may be applicable. ' 

4. Examine the characteristics of enabling knowledges, Subtask 05.22, bear- 
ing in mind the possibility of applying the innovation to these kinds of 
instructional content, as well as to reference knowledges. 

5.. Since reference knowledges are relatively, independent of specific skill 
training courses, or may generalize as paRt content across a wide range 
of skill courses, the assessor may have io proceed directly to the data 
base descriptions of all training courses in the Navy inventory. "Theory 
of Operation" is perhaps the only term which' is fairly standard for this 
content in all courses and curricula, at least in equipment oriented train- 
ing. (But even courses in the culinary arts have a theory of dietary 
. principles, and navigation courses often contain theory of celestial 

mechanics.) This state of affairs places a burden on the assessor'.s access 
to know edgeability about the knowledge content of courses in general, and 
terminologies used for course objectives that apply to inter pretations of 
the applicability of the innovations. 

RATIONALE AND COMMENT 

The issues raised in step 5 make up the genefei difficulty in the classifica- 
tion of training materials that do not quite point directly to skill criteria 
or to a pattern of steps, each one of which has a tangible and identifiable ' 
function in progressing towards a skill objective. The characterizatTSTof 
the skill objective, being subject to evaluation in- an operational environment • 
where work output can have assignable value, simplifies a naming^ technique that 
IS fairly unambiguous. By contrast, the naming of knowledge content may be 
somewhat arbitrary in term^^-of its incl usi veness on degree of detail 
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Th^ assessor may, however, find another dimensioning of knowledge (cognitive) 
materials useful. They are related to R. M. Gagne's taxonomy of building 
blocks in learning (see his Conditions of Learning) . but do not imply 
sequential -progression. They do imply differences both in the learning process 
and i.n effective instructional technique, m brief, the classifications are: 

Facts., prescriptive rules , and other content learned by repetitive rote 
, . practice or memorization". Tables of information, such as the arithmetic 
tables, or one-for-one codes. The rote operations may be embedded in 
memory aids and meaningful contexts, such as the name of a device with a 
picture of description of its use. 

Concepts , which are ideas that have generality, such as of the properties 
of things, or of functional relationships among things or processes, or 
structures. Thus, "hierarchical rank" is a concept, whereas "Always 
salute a superior officer", is a rule. A concept is a basis for making 
equivalent'the members of a class of things, as red and blue are equivalent 
under the concept of "color". Concepts have to do with the meanings of 
abstract terms. They are learned by abstracting the common feature among 
a collection of otherwise'different experiences. 

a 

principles are combinations of concepts in the form of sentences, such as: 
The vehicTe moves (a concept) when its wheels have sufficient friction or 
traction on the roadway (another concept). Principles are learned and 
applied by linking, verbal statements to-a variety of experiences which 
manifest the principle. . ' 

Problem solving may be the appl i cation, Af a structure of facts, rules, and 
principles to the structure of a proBlem. sj tuation. 

These are highly condensed characterizations, each of which justifies a peda- 
gogical chapter to make operational for the purposes of instructional analysis 
and design. It-is possible for innovations to be highly applicable (efficient 
and effective) for one of these classes of learning, but relatively inapplic- 
able to the- others. Thus, much of traditional programmed instruction is directed 
mainly towards the learning of rote content. By definition, items in rote con- 
tent have little context (meaningful associations), attached to them. 



TAEG REPORT NO. 12-3 , 

TASK SUBTASK PAGE 
Q5 21 01 



TASK: DETERMINE RANGE-OF-EFFECT 



SUBTASK: IDENTIFY TYPE OF ENABLING KNOWLEDGE 



DESCRIPTION 

An enablirvg knowledge is one which facilitates the learning of a skill. It 
may or may -not be relevant to the exercise of the skill after the skill has 
been acquired. Although the concept of enabling knowledge has been widel-y 
used as a rationale by educators for teaching almost anything whatever as "pre- 
requisite" to skill training, the, present pragmatic approach to the acquisition 
of on-the-job skills emphasizes a direct relationship between the l/nowledge and 
skill acquisition. The following categories cover the major range/ of enabling 
knowledges that are quite, clearly applica^ble to skill learning. 

An enabling knowledge tends to be most effective when it is acquired imme- 
diately preceding training on the target/ skill , or some phase of it, but the 
economics of instruction may be thought /to justify separation in time and 
instructional context between learning t!he knowledge and learning the skill it 
should support. 

PROCEDURE 

1. Identify the type of enabling knowledge applicable to the, innovation, 
t Operational Goal Criteria i 

The variables in the performalnce of skills that determine goodness of 
the performance. Some of thdse criteria may be imme^liate and local, 
others may be remote in time and locale. In effect,' this is not only 
what good versus poor perforn|iance outcomes look likp, but why they are 
important to the incumbent, tio his organization, aqd to the Navy mission. 
Realistic instruction may include the specification of conditions under 
which, some criterion performance levels may be relaxed as a trade-off 
against no performance at all! If such material is shown in operational 
contexts, perhaps with specifi\c examples, the student may acquire sets 
of values that enable him to ekercise judgment under the pressure of 
conflicting objectives in work ^situations. Verbal abstractions and 
exhortations may be ineffecti ve\in transferring to the work situation. 
In general, learning about operational goal criteria will be useful to 
/ the student in operations, to th^\ extent that he can relate them to 
■ choice^ in his own actions at the\time he must sfelect the action. 

• Nomenclature, Identification, and \ocation of Work Objects 

' The learning of the names of things\associated With tasks, what they 
look like, and where they are located, has direct transfer value to 
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the TWning of tasks that are. not solely manipulative. The better " 
that noiwenclature and identifications are mastered, the more readily 
can the^^udent apprehend and remember instructional content that 
deals with that nomenclature and associated images of appearance and 
location. 

Instruction in this material may recognize that learning is faster 
and retention better when the elements are learned as patterns in 
contexts that are meaningful to the learner. 

t Procedure Descriptions 

Procedures may be described at many levels of abstraction. How 
efficient the learning of abstract descriptions of procedures my be 
to their later learning at the level of detailed steps is not of 
concern here. Presumably, any image that the student acquires about 
actual operating procedures which he will have to perform may have 
some mnemonic value to later learning of the actual procedure as a 
skill. When the description of the procedure to be learned by the 
student is highly task specific, the student is learning what is 
called here a "task format", which is treated more systematically in 
the section of that name, Subtask 05.22. 

Procedures applicable to tasks may extend to action policies, principles, 
and strategies. Instruction here may consist of what these policies and 
principles are, rather than in practicing them in simulations of task 
performance. Knowledge about these policies and strategies may facilitate 
learning them in practice as a skill. 

It should again be underscored that an enabling knowledge as con- 
trasted with a task format — about procedures consists of generalities 
about procedures or protocols, and not the specific learning of the 
procedure as a literal action specification. Thus, if the student 
merely skims a procedure specification in a handbook of instruction, it 
would be appropriate-lia^.c^ll the result of this skimming an enabling 
knowledge. But so would be a general outline of the procedure which 
had to be learned, as such, to a high criterion of recall. The result 
IS a general impression of how the procedure is performed. 

t • Job Relevant Facts 

These are facts and data directly applicable to job*tasks. These range 
from the number of degrees in a compass to the temperature at which a 
soldering flux bonds a given metal, the calories in a pound of potatoes 
or the electrical resistance denoted by various color bands on a 
resistor. Each of these examples can readily be tied in with a class 
of job task. The relevance of a fact to a task can readily be deter- 
mined by asking if the task can be effectively performed in the absence 
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Q°[ven°i^"?hP°HPc'''"i"?' Unfortunately, care is not always 

given in the ,design of training content to differentiating facts essen- 

wen o°ns st'^?'/[p?h''°^' I''' 'l' gratuitous. (An innovation ! 
wen consist bf a technique for making such distinctions.) 

?rSnp'*^''%"r • specific in the identification of task-enabling know- 
other nstru" tJnn^rr'? °l instruction. This content is often m?xed w^th 
identifv wIh^I ^ '"'"^ ^'^^^■"^■"9 courses will explicitly 

,n?f n7- "of^enc ature", or a synonym, as the training objective af some 
P? edure"' n'.'nT; equivalent designations of "Gener 

Procedure in some courses, expecially those for duties at the officer level 
But It may be difficult to parse out course content that is directed specif ^ 
cally towards instruction in "Operational Goal Criteria " The assesso? mav 
require extensive knowledge about courses, or interpret the descrpons of 
their objectives and content in order to identify enabling knowledges 

TJIfp^IT ^^9^^e9ree of precision in estimating the potential 
effects of innovations dealing specifically with enabling knowledge can be 

J^l alternatives are either detailed manual examination of 
ZtPnt JpV' gross estimates of percentages of given course 
content, m terms of student hours, spent on enabling knowledges. 

II;L°nJ''c\?^ f^nc'oo^^''"' P^^formed in this subtask should be passed . 

tS whether thP^ "'i'^ °' '''^ °^der tS check as 

to whether the enabling knowledge supports instruction in task formats.. 
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TASK/ DETERMINE RANGE-OF-EFFECT 



SUBTASK: IDENTIFY TYPE OF TASKJORMAT 



DESCRIPTION 

A task format is a structure of how the task is done. One can learn the task 
format without being able to perform the task in real life circumstances. But 
if the task format is genuinely applicable, the learning of the format will 
generally facilitate learning the task as a skill. This does not assert that 
learning, the format at the same time that one practices the skill will not be 
more efficient than the two-stage process. And the way in which a format is 
learned may be signi ficant in its value to learning th--? skill. 

Task formats may be general or they may be specific. Thus, the idealized 
general format for using a test instrument consists of: (a) determine the 
variable to be measured, the range of the variable to be measured, and the 
degree of precision in measurement; (b) locate the test instrument and identify 
it by name, and check its function and capability according to the purposes of 
the moment; (c) check its calibration . . ., and so on through its use and 
return. These are conceptualized set of steps that apply to the use of any 
test instrument. It is an example of a task format. 

At another -level , a task format may be a verbalized specification of every 
detailed step to take in selecting and operating a specific voltmeter. The 
verbalized instructions are not the equivalent to the ability to execute the 
self-instructions in practice, but they may serve as a guiding template for 
carrying out the action. 

A printed requisition form embodies the task format of preparing a requisition 
for one or more objects or services. The names of the. blanks to be filled in 
are the variables in the transaction. The entire task format for making a 
requisition may include finding a copy of the form, filling it in, and trans- 
mitting the order form to the purchase agent. 

The learning of a format enables a person to copy with" a variety of different 
situations that fit the variables that make up the format. The concept of 
task format (whatever the concept may be called) is important to the economics 
and practices of training and transfer of training. It is not a novel notion 
in educational and training practice^ but its role in training technology should 
be made ex|)licit. -Instruction in task formats is relatively inexpensive in 
training equipment because it can generally be accomplished with symbolic exer- 
cises.. The sections in this report that deal with .short-term memory and with 
decision-making (Appendix A on Task Structure Elements) amplify the meaning 
and application of the task format concept. Appendix A also contains an analytic 
essay on task formats. 
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E^ablU^':owlIdge'/'''^ ''''''''' is set apart from the topic 



PROCEDURE 



Detennine whether the innovation is aimed at instruction in some one or 
another type of task format. This decision will be based Sn the Senegal 

tlZuT lL'rn\l°r''°'''r' '''^ ^^^^^^ underlsCRI t' on 
Append X A, and in the topics under step 2. If the innovation is 

d^re^^lirto^siStalfoB^^T^^ '''''''' de^cS^KeJ^ go on 

?hP^^i;nL-^^ ^^P^""^ f^P" °^ which the innovation appli^es 

I?ve kr"n^/r l'''^^ exhaustive of fon.at types with respec Eo cog " 
tive tasks, or tasks primarily directed by cognitive processes. 

t Procedure Formats . ' 

» 

A task format may serve any one or combination of the following 
Inn^ h°"5'- ^ structure for searching for task information; for 
apprehending and retaimnc? task information in working or short- 

^0% Dres'si^a LfFf^ lonVterm- memory : 

SquSlf^e fa'k ! " ' '"^ exe^cutlM the'^work output^ 

?U^nf task formats may apply to any one or combina- 

tion of Task Structure Elements Subtask 05.13). A given work 
procedure may emphasize some one of these task elements burmanv ^ 

f°forLT fu:''l\°\' '''''' in s.equence At ne le el 

;h/°r '•*^!u'*^'*^"* "memorizes the verbal statements that comprise' 

support the first - is his memorization of a set of pictorial 

ic?'re 'nf\hP ' '''I'' °' '''''''''' ^^^^es, or'a mo? o 
picture) of the sequence of steps. Another support may be 

nhStr'*-T°"L°^ procedure by the instructor with actual work 
r.i . ' J° these various "encodings" of the d?o- * ' 

• ' m he sSden?' '''"'/^'^S 't'' '^^y strengthen each otSe? ' 

to lrLli£ ly. f ""^^ procedure, and his potential ability 

Task formats may be learned to some useful degree, independent of 

sit'uaJinnr '5%^^ with real work ob ects n re^^^^ 

situations. But the learning of task formats is not necessarily 
independent of learning the task by direct practicroS it Bevond 
given levels of learning on a give^ task format, fl ?s ikely 
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additional practice on the task format will have diminishing --turns 
in transfer and savings effects on the learning of the- live task." 
The smaller the demand by the task on complex information processind^ 
by "the student, the smaller the relative amount of practice on the 
task format that will be useful. Sequential procedures where each 
step is a relatively simple situation-response unit demands, by 
definition, relatively little information processing except that 
of the sequence of steps. ' 

In conclusion, many procedure formats will, in practical terms, consist 
ot the verbalization and imagery of. procedural steps that comprise the" 
task The 'rules of play" for the game in chess is an example of one 
level of task format for the game. 

Decision Formats 

Because decisions are information processing operations, decision 
formats and exercises in decision-making can be accomplished. apart 
from live task settings.. There are exceptions, such as when the 
decision is a progressive one and depends on interaction between the 
incumbent's overt behavior and the response of the.environment to his 
behavior. But even in^such cases, limited decision formats may use- 
fully be learned independent of live practice. In fact, this is what 
many instructors attempt to do when briefing a student in advance of 
a maneuver in what he should attend to and be prepared to do. The 
briefing establishes at least a temporary format for the performance. 

A decision format is primarily a transactional template of (a) the 
variables that make up a given type of problem situation, (b) the 
variables in the relevant response set from which an alternative is 
to be chosen, aTid (c) the rules, policies, or strategies for select- 




.,, the 

given case. The once popular business-gaming exercises attempted to 
teach students of business the„formats of marketing decisions. 

Strategy principles for selecting a move, or a series of moves, in 
chess, are an example of relatively complex decision formats above 
the level of rules for moving chess pieces. 

Construction Formats 

Whereas, decision-making tends to be a process of converging on the 
selection of a response from a limited response set, constructing 
tends to be both a divergent a's well as convergent process. The 
design of a pattern, such as a plan for building a bridge or for an 
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attack mission, often requires the construction of alternatives as 
well as their selection. 

Specify the type of format, and make a judgment as to the range of types 
of job tasks for which the innovation might be applicable. Note that all 
requirements for verbalized learning of procedures in order to fulfill 
verbal tests of procedural competence, point to what is here called a 
"procedure format." Any training objective which reads: "Must give a 
detailed verbal description of the procedure," identifies the learning 
of a format in the sense applicable here. Training in decision and con- 
struction formats is less well understood and practiced. 

Key terms applied to job tasks, and training courses that may be appli- 
cable to the innovation applied to decision-making and constructions 
include; troubleshooting, diagnosis, fault localizing, "planning, schedul- 
ing, laying out, constructing, problem solving, calculating, and plotting 

Pass the descriptions generated in this subtask, as well as information 
coming from Subtask 05.21, Identify Type of Enabling Knowledge, into 
Subtask 05.14, Identify Relevant Jobs and Tasks. 

The alternative can be to. use the descriptions of the task formats, 
applicable to the innovation, for going directly to the data base of 
course description,, bypassing the job task analytic route. 



vr 



111-15^ 



TAEG REPORT NO. 12-3 



TASK SUBTASK PAGE 



05 



23 



01 



TASK: 



DETERMINE RANGE-OF-EFFECT 



SUBTASK: 



ASSESS ON-THE-JOB BENEFITS 



'DESCRIPTION^ . . 

The making of rigorous cost-benefit assessment of improvements in on-the-job 
performances of various kinds follows the same principles that were used for 
developing and applying to functions in the training enterprise. But the gap 
between principle and practice has not been bridged for formal on-the-job ; 



However, lack of formalized analytic techniques should not deter useful judg- 
ments to be made in practical circumstances where something needs to be done, 
and an informed guess is better than nothing at all. 

The innovation may not be justified by benefits achieved solely within the. 
training organization, such as by reduction in displaceable costs in instruc- 
tional vehicles, reduction in training tiine, or reduction in aptitude levels. 
In consequence, the justification of the innovation will depend on its making 
a worthwhile contribution to graduates doing their jobs in actual job 
environments. . 

A rigorous cost justification of increased job competences derived from analysis 
in the operational environment is ra)*ely feasible, and when it is, critical 
variables will be omitted. Examples of such variables may be morale, depend- 
ability, readiness, initiative, dedication, and similar abstractions which 
although having subjective meaning and sometimes even predictive validity when 
based on subjective evaluations, fail in ob-tiecti vi ty necessary for cost and 
value analysis. 

The following is a nonexhausti ve list of types of on-the-job benefits which may 
result from innovations that "improve" the product of training: onrthe-job 
skills and competences. It is assumed that the innovation^ has been put through 
the range-of-effect analysis, and that tasks and jobs relevant to the innovation 
have been identified quite specifically. From this identification, the number ' 
of incumbents to which the innovation is now, or potentially, applicable can be 
quantified. Having identified, the one or more types of benefit applicable to ' 
the result of the innovation in a job skill, the assessor follows the same format 
for describing benefits as outlined in 05.18. 

In this case, the assessor is not likely to be a training officer, but an opera- 
tions officer familiar with the relevant job skills and the work environments in 
which they are performed. Only such individuals are able to make useful judg- 
ments of magnitudes and importance of given enhancements to specific skills, 
and the probability of realizing the benefit under actual work conditions, Vhus, 
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if a person's duties, as they exist, tend to demand his full time and attention, 
there is little benefit to be gained from giving him skills which demand 
additional time, or that require him to be elsewhere. Nor is there point to 
providing him with a skill to be exercised where the tools necessary for exer- 
cising the skill are unavailable. A potential benefit must have an opportunity 
to happen in order for it to become an actual benefit. 

Having completed the analysis of a potential benefit derived from an innovation, 
^an attempt may be made to'determine its value in various kinds of savings in 
operational environments, such that these savings can be translated into dollars. 
Examples of such savings may be in term? of: 

Reduction of manpower units in j:he operating Wganization. 

Reduction in tools, supplies. 

Reduction in expendable materials wasted per unit of mission 
effectiveness; e.g., wasted rounds per kill. 

Reduction in units of standby equipments for operational' readiness. 

Reduction in probability of catastrophic failures; e.g. , personnel 
casualties, planes loist, ship's engines crippled, ' coll ision at sea, 
major oil spill, nuclear diffusion to open environment. 

But where savings of this kind cannot be detenninetil or are inadequate to account 
for the benefit, a subjective evaluation of the benefit must be made by responsi- 
ble officers. This evaluation of the benefit must then be weighed, subjectively, 
against the costs of the innovation and the probabilities of its being realized. 

If the added cost to training incurred by the innovation must \vork within fixed ' 
training and other personnel resources, then, the evaluating officer may have to 
make a second order decision: what to sacrifice in the existing products of 
training to compensate for the added cost of the innovation. This kind of 
decision is typical of management that operates under the constraints of budgets. 

The following are types of benefits from enhanced skills'. 

Greater .flexibility in assignment. 

Less on-the-job training required for acceptable work performance. 

Higher productivity levels, higher quality of performance (fewer errors).- 

Greater range in work competence in situations, environments. 

Willingness under stress -- increased motivations to continue working 
under adverse conditions of discomfort, hazard, fatigue. 
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' Initiative: increased capability for self-initiated and self-directed 
activities. 

The use of one or more of these descriptive titles should be augmented with 
specifics related to tasks and task contexts in the description of the benefit. 

PROCEDURE 

1. The assessment of the benefit should be completed by using the "Format for 
Describing a Supplementary Benefit or Liability" in Subtask 05.15. 
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TASK: DETERMINE RANGE-OF-EFFECT • ' 

SUBTASK^: IDENTIFY RELEVANT FUNCTION IN DEVELOPMENT AND ADMINISTRATIVE 
\ . MANAGEMENT OF TRAINING 



DESCRIPTION ■ ^ . .. 

Thegivin^ of instruction is preceded and surrounded by many kinds of support 
activities^ Although some of these activities may be peripheral to training 
• functions, Jiiany of them are mainstream arid essential. These activities can be 
clustered into nodes or functional tasks in the support of training. As pro- 
cessing functions, they receive, process, and output decisions and other classes 
of work infor;'mation. 

Much of the information developed as output from one node of task activity in 
the training enterprise, is passed along as input to another node. Thus, task 
analysis and jt?b requirements and specifications, as an output, become partial 
input to another node which develops "course specifications" as an output. ' 
This, in turn, jpasses into a node which generates "course content analysis and 
development." Finally, this sequential network, with contributory branches, 
results in instruction to students which, hopefully, results in learned skills 
which in turn result in on-the-job performance. ' ' 

With some exceptions, this network of noaes seems rather loosely coupled. That 
IS, kinds and categories of information that function mode A should produce as 
input to node B in-order to precisely meet the requirements of functidn B, is 
generally vague. Thus, much of what B ought to have, it does, not get from A 
And much of what it does get from A, it doesn't need or know how to use. B's 
.difficulties stejn partly from not being tightly coupled to function node C B 
does not know precisely what input C needs to do its job, and this uncertainty 
IS transferred Erack ^to the relationship of B to A. , 

This loose coupling of information structures and relationships presents great 
problems in any attempt ,to evaluate an innovation which claims to "improve" the 
effectiveness of node, function B. If the innovation merely claims -to do what 
B does now, and no better or no worse in terms of inpjt demand or output product, 
then the innovation can be evaluated only on whether it saves costs in perform- 
ing or supporting function B. If reducing the cost of the B operation imposes 
a greater cost on what A must produce, however, then the savings- must be the 
net from the savings effected in B, minus the increased costs in A. 

But suppose that the innovation claims to improve the "quality" of B's output 
to function C, but at an increase in the cost of B. The benefit must be 
assessed by what happens in C. If Cs output to D is not improved as a direct 
result of B s improvement, the claim of improvement in B cannot reasonably be 
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substantiated, with this exception: the cost of function C is reduced as a 
consequence of the new. kind of output from B." This chain of referral of 
'benefit" ultimately ertds in the performance- of the student after he gets on 

assessing improvements in that performance that can feed back 
ta justify, the increase in the cost of function B. The more tightly coupled 
the functional nodes are to each other, the more readily can such evaluation " 
be made^. . . 

This observation should make it clear why many innovations that have high face 
validity for improving a. function in the training subsystem, are virtually 
doomed unless there are corresponding changes all along the line, or unless 
the innovation really eliminates duplicated work among various nodes ^- but 
this should really b.e evaluated as a cost saving, an improvement in the 
efficiency of what is done now, and not an improvement in the quality of the ' 
service.' It should be noted that in actual training operations there is a 
common reference criterion of "quality": how much did the student learn and 
perform per unit of.-cost? .This ultimate criterion is measurable ,. hence subject 
to assessment against a cost base. . " 

' Intuitively at least, there seems ■ to be massive potential for improving the 
qual-ity and cost picture in the many supporting activities that -are here called 
Development and 'Administrative Management of Training" fD&A) . ..But without the 
ability to trace tfirough the improvement in quality of output in a given D&A 
function to a knoW-ledge or skill product, the assessment of changes having the 
intent to reflect in improved quality of operations must be Limited to demon- 
strations of cost improvements, or to quite intuitive hunches that somehow 
better training will be the result. But, because the system can readily absorb 
Local changes without overt effects, the hunches are likely to be overestimates. 

A process model, therefore, needs to be developed which, at some practical 
level of precision, can encfbl^ an innovation in technique or tool made in 
function B not only to be measured in cos,ts to B and alT of the functions that 
, feed or are fed by B, but also to relate the change in B to an ultimate quality 
benefit in student skills. It is naive to assume thafN^uch a process model can 
be as rigorous as a computer algorithm, but it need not'be. Part of the art in 
developing such a model would be knowing when to stop in seeking precision, and 
when a practical level of precision had been reached. 

These considerations, therefore ,""Tead 'to" focus on reducing costs i-n given opera- 
tions Within a D&A node, or among- a group of nodes. Such analysis cannot 
differentiate whether the reduction in cost is in something which affects a 
training product or is irrelevant to a training product. In the case of an 
exception, where the assessor can relate an innovation in a function directly 
to a learned product in on-the-job performance, his assessment of the latter 
•c ul identify the Learning Objective, and proceed through 

bubtasks Q5.11, 05.12, and so on, through 05.18, in the task content training ' 
path'. • ^ 
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f?Jlt^'oi^xhaust'i°vr°^J/;;''^°"' ''''' characterizations of them. 
* 4.U IS not exhaustive, and the innovation may not fit anv Dartimiar nno 
of them. This is not- particularly important, becLsrin any event Jhe Lessor 
would have to localize the processes or products in the D&A network which ?hP 
innovation'was specifically intended to replace or displace. 

Capture and Description of Skill and' Knowledge Requirements 

I?!^ J-!,?^set of techniques. Their application is of variable qua! itv 
V Jtna V ^I'l' invariably time-consum-ing. The data feed' n o 

virtually all training operation^ that follow and extend into iob 

• Matching the Skill Requirement With Descriptions of Available Resources . 

' ♦ 

This is a decision-making situation. The operation may consist of' 

• Hf^\'^ ";atching a pattern of skills requirement with existinq 
skill and knowledge in trained and/or experienced persons; or it may 

- be an attempt to match the requirements with a pattern of existing 

resources in the form of aptitudessto learn the required skills ■ / 

/ 

• Matching the Skill Requirements With Available Training Resources / 

I5iL'%LniHlr;n'h'"^ ^"'■"^'■"9 resources consTst of/, 

space, facilities to house and teach, existing courses, and person/el. 

• : Developing Training Requirements From Skill Requirements Descrfpt^'ons . 

A set of techniques. The training requirements picture will be^a 
breakdown of the overall skill requi rements into enabl inland 'terminal 
instructional objectives,, derivations of knowledges to support sH Is 

'/k^^J^* functional requirements of instructional ve es 
and allowable training time per student. uij'a i vkm i c i es , 

■.• Selecting the Training Environments / ' 

JeiuiSenis" '"u luf^^ o^/o^ow the specification o/the training ' 
requirements. It will result in a choice of what train/nq obiective^ 

?r.'in?nS'/H^'^''^.^"^°^"^'^ ^^^^"^"9' and which ones in on-t^^^^ 

• ^p"l•s:?s^sSa?^a^b^T""• ^^^^^^ -^-^ thf 

• Slow Versus Fast-paced Development of Training Material 

t 

This is. a decision which is followed by .technique. On occasion the* 
demand may arise for getting people trained at a very early da'te sa 
that the normal processes in the development of trailing must be bj- " 



ERIC , • ■ 



TAEG REPORT NO. 12-3 

I 



TASK SUBTASK PAGE 
'05 24 04 

passed or accelerated. This bypassing may be costly in hard-to-find 
talent and other training resources'. 

Selection or Devising of Training Modules 

A decision followed by techniques to implement. A training module may 
be a course used as an element in a pattern of other C9urses. The ' 
module thus serves more than one purpose. The module may be instru^c- 
tional entities within courses or even independent of^courses. A good 
modularity is one which enables the smallest set of modules to be 
organized to serve a wide variety of purposes with a minimum of waste. 

Selection of Instructional Vehicles and Media 

A decision-making operation. The result is hot merely the choice of 
individual vehicles, but a set of them which, together, are cost- 
effective in producing a pattern of interlocking skills and knowledge 
making up a duty, job, or on-the-job competence. 

Designing the Content of Instruction 

Techniques for developing the content of what the instructional vehicles 
deliver to the student. This results in what the student actually sees, 
tries to learn, and does. The technique may be general to any training, 
or restricted to a type of skill or knowledge objective. If the latter, 
further analysis of range-of-effect should be made. by following the 
procedures in Subtasks 05.19, 05. 20, /and 05.21. 

'Devising Selection Procedures for Students 

Techniques for relating- the level and range of training objectives for a 
job or duty, and time allowed for the training to these objectives with 
.entrant capabilities in the form of existing levels of skill and 
knowledge acquired by training or on the job, or in the form of measured 
aptitudes predictive of learning rates and potential terminal skill 
levels. Note that this item relates to the previous topic on "rMatching 
the skill requirement with descriptions of available skill resources.'' 

Assessment of the Student ' 

Techniques for efficiently sampling from student skills and knowledge at 
the end of training, with respect to training objectives and job 
requirements. Also, decisions as to cutoff levels for washouts. 

Assignnient of Graduates to Job Positions 

Techniques and policies for matching the competences (or the description 
of competences) of graduating students with the identification or. 
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description of job and work environment assignments. 

• Evaluation on the Job 1 

Techniques for efficiently and effectively asjiessing the on-the-job 
competences of the graduate, the extent to wjiich his range and level 
of [Performance is unacceptable, acceptable, or superior, and the 
transcription of these data on the student's -record for ffi|se in his 
later training or assignment, and for use in 'the revisiotiHof training. 

• Career Development ; 

Techniques for the organization of a progression of training and'gob . 
experiences that are psychologically cumulative and operationally 
practicable, and decision structures for the matching of a career 
p^attern with individuals at various stages of training and experience. 

• Administrative Planning and Control 

Decisions and techniques for the planning and control of the overall 
operations of the training enterprise, or substantial segments of it. 
The adapting and regulation of input requirements and objectives with 
existing and planned resources, according to organizational policies 
and values. The information base may consist not only of current 
status and projections of immediate future status, but archival data 
that enable experience to project the implications of trends, and 
the range of variability in system behavior within which the perturba- 
tions will damp themselves out, but outside of which the perturbation 
will amplify itself. The information base may also contain long-term 

^ future predictions of needs that enable the application of structural 

I models for developing game-playing solutions. 

• Evaluation of Learning or Entropy of the Training System 

Techniques for assessing and evaluating whether the rate of adaptive' 
change by the training enterprise as an operational str-jcture is' 
progressively losing control of its overhead costs, the relation of 
process to product, and to the required quality of the product, or 
whether the rate of adaptation is outpacing the entropy characteristic 
of very large organizations over a period of time. This functiorl is 
somewhat outside that of operational administration. 



PROCEDURE 



Using the foregoing categories as a guide, but without being limited to the^ 
subject matter identified by them for D&A functions, identify the tasks and 
task activities to which the innovation is relevant, both as a benefit and 
as a liability. The relevant tasks should be' identified as precisely as is 
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practical for analytic assessment of benefits and costs to be derived 
from the innovation. 

COMMENT 

A useful concept for characterizing and bounding a task consists in making 

a "transactional structure" of the task. This can be done at any of many 

levels of detail or generality. The structure consists of the following: 

a. Source of input information to the task-function. 

b. Variables in the essential input "message format" that feed into the 
function. One or more input message contexts become a "demand" for 
service from the task-function. 

c. Variables in the essential output information or message format from 
the task-function the output represents the "service" performed by 
the task. 

d. Variables in the reference information that serve as supplemental 
input to the task-function.. 

e. The operations > principles, procedures, rules, and problem-solving 
techniques that combine input message contexts and reference informa- 
tion, and transform them into output messages which represent the 
service performed by the function. 

Notice that the basic characterizations of input, output, and reference 
information are in the variables or categories of information that comprise 
a message context, whether of an input demand for service, or reference 
data for generating into output, or in the output information that is used 
as a service. The categorical structure of the inputs to and outputs from 
a task enable the tracing of functional continuities and relevances from 
one task to another in a sequential network. This coupling enables- the 
effects of changes in one part of the network to be predicted (hence, 
assessed) in other parts of the network. 

2. Pass the information developed in step 1 on D&A tasks relevant to the 
innovation to Subtask 05.J8, Assess the Training Benefit Pattern. 

COMMENT 

The foregoing descriptions of nodes and task-functions of D&A activities 
are essentially the descriptions of human tasks, since they are performed 
•by job incumbents, and supervised and monitored by humans with task 
responsibilities. Procedural step 1 will have identified the job tasks 
in D&A operations to which the innovation applies. 
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The structure for assessing the benefits of on-the-job tasks is presented 
in Subtask 05.23* in the section dealing with benefits that an innovation 
which leads to improvement in a skill or competence can have for the job 
environment, Techniques for the quantitative analysis and evaluation of 
improvements" in on-the-job functions are as applicable to D&A jobs as 
they are to the jobs dealing with the operation and maintenance of equip- 
ment, or in providing clerical or professional services. 
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TASK: PERF'ORM COST-BENEFITS ANALYSIS 

Task_3 (Make Preliminary Feasibility Profile) and Task 4 (Perform Analytic 
Feasibility Assessment) were concerned with preliminary assessment exercises 
o on limited range-of-appl i cation data. Based upon a tentative decision to 
accept the proposed innovation, Task 5 (Determine Range-of-Effect) will have 
identified the full range of applications and will have, also, provided the 
full pattern^of benefits (and. liabilities) intrinsic to the innovation. The 
purpose of this task is to incorporate these two additional sets of informa- 
tion within the cjecision process. The decision tree used in Task 4 will be 
revised to include: 



Cost data, derived within this task will be used in Task 7 (Perform Financial 
Analysis). It should be pointed out that two separate decision patterns 
are being developed to assist in the ultimate decision-making step. The 
first uses the decisi on "tree where an attempt is made to quantify benefits 
(including dollar savings) and liabilities (including dollar costs) and then 
to develop a value for each 'of the possible decision paths. This approach 
will aid the overall decision process, especially when normally nonquantified 
benefits and liabilities will have a substantial influence on the decision 
maker's view of the overall acceptability of the proposed innovation. The 
second decision pattern (outlined in Task 7) is concerned with a straight- 
forward financial analysis of the proposal using identifiable costs and savings 
^ over an extended time horizon. There the decision criteria will be such 
factors as net cost/savings and return on investment. The nondollar benefits 
and liabilities addressed in this task (Task 6) will not be included in the 
financial analysis. However, the results of the two distinct decision pro- 
cesses will be presented to the decision maker as outlined in Task 8 (Make 
Accept/Reject/Study Decision). The major subtasks within Task 6 are out- 
lined below. 



More precise cost/savings data for each outcome 



Benefits/liabilities data stated in equivalent dollars 

Refined success probability es'timates 

A sensitivity analysis identifying critical variables. 




SUBTASK 



REFERENCE 



the newly acquired rqinge-of-effect and benefits data 
from Task 5. 



a from the preceding tasks including 



06.01 .01 



2. Prepare appropriate caSes and cost model input data 
for each decision tree outcome. 



06.02.01 
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Run cost model for each case to determine dollar value . 06.03.01 
of each decision tree outcome. 

Determine equivalent dollar value for each of. the benefit/ 06.04.01 
liability patterns. 

Refine the success probability estimates considering any 04.02.01 / 
additional risks frqm any extended application. indicated 04.04.01 
from Task 5. 

Develop new decision tree assessment diagram using the 06.06;01 
parameters developed in -the preceding subtasks; then 
determine new decision values. - 

Perform sensitivity analysis of decision path to 06.07.01 
selected variables. 

Create tickler file entry identifying criteria for re- 04.06.01 
assessment of innovation if, at this point, the proposal 
does not appear feasible. 
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TASK: PERFORM COST-BENEFITS ANALYSIS 



SUBTASK: ASSEMBLE 'DATA 



DESCRIPTION 

The cost-benefits analysis further develops the decision tree assessment for- 
mat from Task 4. These data, along with the full range of entities (courses, 
jobs, equipment, etc.) and benefit characteristics identified in Task 5, will 
form the basis for the defined cost-benefits analysis of Task 6, 

PROCEDURE 

1. Obtain the final decision tree prepared in 04,05 along with rationale 
supporting the separation of costs/savings by outcome and the probabili- 
ties assigned to implementation and user acceptance success. 

2. Obtain the final full range of potential applications for the innovation 
from Task 5. These will include,, where appropriate: 

t Course numbers (CDP) • -.^ 

• Job designators (Rate, NEC, NOBC, etc.) 
t Equipment types 

• Student types o 

• Cost elements 

• Taxonomic search parameters 

t Other directly costable elements. 



3. Obtain the allocation rules wfTich identify how the pattern of benefits/ 
. liabilities is to be apportioned across each of the entities selected 

as being potentially impacted by the proposed innovation. 

COMMENT: Typical allocation rules might be-: 1) 10% of the Vaboratory time 
in couf-se 12AB will be saved; 2) one hour of maintenance per month will be 
saved on the XXXX task ctrainer; 3) three weeks of OJT time for ET2's and - 
ET3's with the following NEC's assigned to shipboard duty will be saved. 

4. Obtain the full benefit/liabilities pattern developed in Tas^k 5 which 
will i^nclude: ^ 

t Specnte^levaiW variable impacted by the application of the 
innovation^ e.g., time to train, attrition, etc. 
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• A figure indicating the relative importance of each variable as 
compared to all relevant ones; e.g., crucial importance = 10, 
trivial importance = 1, etc. . 

• Magnitude and direction of change in each relevant variable; e.g., 
attrition reduced from 20% to 18% or attrition reduded 10%. 

• A confidence range for the estimate of each relevant variable; e-.g.-, 
under the nrost adverse conditions, the most likely estimate of 
attrition (18%) will be no greater than 20% and under the most ' 
favorable conditions, it could go as low as 15%, or the most likely 
attrition reduction' estimate of 10% may vary from 0% (worst case) 

to 25% (best case). 

5. Derive the expected value of the relevant variables. Since the variables 
will generally be bounded at both ends by practical limits and these 
limits are expressed by the confidence range identified as a part of 
the benefits/liabilities pattern developed in Task 5, a useful estimate 
of the expected value (v^) is the PERT statistic:* 

■ «■ • 

vp = p + 4m + 0 ■ 

^-6 . 

Where: = expected value of the relevant variable 

p = most pessimistic value of the variable 

m = most likely estimate of the variable 

0 = most optimistic estimate of the variable. 

The value derived will be used as the cost model and the decision- tree inputs, 

COMMENT: The assessor may want to review 06.07 (Perform Sensitivity Analysis) 
to become familiar with the further use of the expected value and confidence 
range. Also, some degree of caution should be used in^this expected value. 1 
estimate, especially when the distribution using p, m,''and o is highly skewed. 
If it appears that the decision will be sensitive to certain variables, more, 
systematic methods are available and may be appropriate for obtaining, the ex- 
pected value and distribution for the variables (see Handbook for Decision m 
Analysis -, Chapter 9). . , . 

(• . ■ ■ 

I . 

* hann, beraid J. ^and Samuel S. Shapiro, STATISTICAL MODELS IN ENGIN EERING. 
John Wiley & Sons, Inc., New Yorky/ 1967, p. 133. ' 

Battersby, A. , NETWORK ANALYSIS FOR PLANNING AND SCHEDULING, St. Martin 
Press, Inc., New^ York, 1964, p. 174. ■■ ~ , 
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TASK: PERFORM COST-BENEFITS ANALYSIS 

SUBTASK: PREPARE COST MODEL CASES AND INPUT DATA 



..DESCRIPTION 

Four basic outcomes (,A through D) have been defined on the decision tree 
assessment diagram. These range from acceptance, successful implementation, 
and user acceptance to rejection of the proposal. (This is not necessarily 
the range considering, the net of costs and savings.) An alternative case must 
be established for each of the possible outcomes which, in the case of failure 
means the formulation of a recovery strategy to which certain costs will be 
attached. Each defined cas.e must have the appropriate cost model and input 
variables specified. 

PROCEDURE ■ " 

1.. Develop basic scenari os . for each of the possible decision tree outcome^. ' 
In the case vyhere the. proposed innovation is accepted, implemented, and* 
used over an extended pUnning horizon, basic development, installation, 
and operational factors will be considered. If -contingency plans will 
involve identifiable cost elements, these should be included in the 
scenarios. 

The scenario for outcome B, where the innovation is successfully imple- 
mented but fails to gain user acceptance over a reasonable period of 
time, might contain the following assumptions. 

Initial planning'TS^t-te same as outcome A. ." ' 

A two-year period will' be considered as a norm to determine acceptance.' 

The Fi)ea.sures of user acceptance planned for outcome A will be -incorpo- 
rated, except that measurement cost will increase by ^0% due to in- 
creased sampling as measurements move toward the control limits. ' ' 



f 
§ 



§ If rejection is indicated at the end of two- years, the baseline plan 
will be' used for recovery purposes. 

0 

t Equipments purchased under the proposed initial plan will have -no re- 
maining value due to their specialized character. 

A similar scenario should be developed for outcome C. Outcome D, to reject 
the innovation, will generally be considered neutral. 
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Prepare cost model input data sheets specifying the values of eacfj variabl 
under each outcome outlined by the preceding scenarios. Input data sheets 
should be prepared for each model appropriate to the costing e^certiseT^i .e 
the training* cost model will accept, for example » the following factors. 

• , Technical factors . ' 

Annual requirements for course graduates 

Attrition rate 

Course length 

Student recycle rate (percent of students set back) 
Average recycle tiipe (time added by being set back) 
- Average equipment downtime 
Instructor/student ratio 
Administrative/student ratio 
Equipment/student ratio 



Facilities^square feet/student 

Facilities square feet/instructors- 

F,aciliti§s square feet/adminis|:rati ve 

Instructional materials development ratio 

Amount of course material requiring new development 
- - P^ercent annual cours.e maintenance requirements 

Supplies/student - ^ - 

Miscellaneous requirements/student. 
Cost Factors ^ , 

Student salary 

Student travel 

Instructor salary ' " , * 

Administrative salary . 

^ • * 
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Equipment unit costs ' 

Life of equipment ^.j, \ 

Purchase policy '"^"""'''^^^ 

- 'Depreciation policy 

- Operation and maintenance costs for equipment « 
Cost/square foot of space * - 
Instructional materials development cost ^ 

- Value of in^ll^iictional materials at end,,bf planning period 

Supply costs/student and student positipn . ' . 

Miscellaneous costs/student and student p,psiti on. 

;he job model may be developed to accept, for example; the following 
actors. ^ 

Annual incumbents affected _ ' 

Job classification (Rate, NEC, etc.) 
Years service 
' Time in grade 
Support personnel ratio ■ ^ 
Job support space (facilities square feet) 
Tdbls unit cost ^ 
Equipment unit tost 
Salary rate (incumbent and support) 

Recruiting/induction costs . . ' 

School training costs 
Transportation costs 
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© Operational time utilized 
0 . Unutilized time cost 
Error rate ' * 
Error cost. 

The equipment model may be developed to accept, for example, essentially 
•^.he equipment related items within the training cost model. 

Numbers' of equipment 

Unit cost of equipment " -..^^ 

Operatkfns' and maintenance cost 
9 Life of equipment / 
» Purchase policy 
9 Depreciation policy i 
• Utilization, 

COMMENT: Each cost model when . developed will have specific formats for the 
input data. I The approach planned for the training cost model is to define 
absolute rather than incremental cos t factors . This will require that the 
input data for the baseline plan be defined* also, .and the incremental value 
of output variables must be Qbta.ined by taking the difference between the 
two runs. Also, costs shouTO be classified along an additional dimension to 
avoid any major restructuring 1n Task 7. The cost categories foir grouping 
data elements are 1) R&D, 2) investment, and 3) recurring, 

3.* Ensure that the input variables reflect the true "opportunity cost" re- 
sulting from the proposal Application. For example, if three weeks of OJT 
time were defined as being tsaved by the proposed innovation, the full 
salary costs or the full t\\ree weeks should only be taken as a savings if 
there is an alternate opporjtunity to use the time saved at the value 
assigned. This should be considered over the entire planning period 
\ under consideration, since ^Hile there may be no value for the time 
initially, additional worklpad in subsequent years may have^value. The 
value for. the entire plannihg period should be factored into the input 
variables. 
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TA^K: PERFORM COST-BENEFITS ANALYSIS 

/ 

/uBTASK: RUN APPROPRIATE^COST M€DELS 



DESCRIPTION • 

f' ■ - . • 

The following outlines the basic steps in making the cost/savings calculation"- 

pr^ ed na1ub?L' : w?iTh' ""''''.'"'^ ^'^''^'^ ^""^^ data from the ; : , ' 
pf^cessing (ADP) ■ P^o.cessed using^^pm| form of Automated D^t^.... .,. 

PROCEDURE ' r i ■ 

^" fh^^^h^"'^"^ ^^^^^^ ^° ^^^""^^ relevant variables have been .incorporated, 
that they are adjusted to reflect the tt=ue economic" va-lue (opportunity cost) 
achieved by aDp|^ing the innovation, that is, the value sacrificed by the 
resource not betng available for some alternative use. If. for example a 

■ certain ambunt of building space is^unus'ed. then it has essentially no * ^ 
snurr; fiTt^' there were a requirement for the space from some other 
source, and that source valued the space at its existing cost, thfen. the 
, full cost would be used. • •■ . ^ * 



..2r 



Training Cost Model. The development- of. training costs will gen(^rally be 
•perfonried on a course by course, or (irogram, basis. The major cSst cate- 

5 general^procedural steps for calculating these categories 
are outlined on the fol lowing" pages . i-cyur leb ^ 

f D^'termine Student Requirements. Given the annual re'quirements for ' 

'^"d^"^ ^"P"^ and student load considering 
Ifength '^3te,^recycle rate, average recycle-time. and course ■ 

* 'hL^r7;!nR?^"'*'?^ PositimRequi,rements. Given the student average-on- 
board_(AOB), calculate the numfaef of Student positions required con- 
sidering the time a student position may be unavailable (e.g.. for ' 
■forJuSueing)^"'* the percent of extra student positions requi red (e.g. . 

t Determine Personnel Requirements. Given the number of student positions 
calculate instructor personnel required considering the inst?uctSr to ^ 
rJn.?5L?°''Jh°" calculate administrative personnel required 

considering the administrator to student position ratio. ' 
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Determine Equipment Required. Given the number of student positions, 
calculate equipments required considering the equipment to student 
position ratio. The actual equipment acquired will be a function of 
its life in relation to the planning horizon, and al'So of the 
particular purchase policy selected; e.g., purchase only to 80% of the 
maximum requirement over the planning period. 

Determine Facilities Required. Given the number of student positions, 
calculate total facilities required considering the square feet per 
student position for students, instructors, and administrators. 

Determine Instructional Materials Development/Maintenance Required. 
Calculate the amount of time for developing initial course materials 
considering the course length, the percent of the course requiring 
development, and the development ratio. Given the amount of time for 
developing initial course materials, calculate the annual maintenance 
requirements considering the percent of initial development time to 
be expended annually for maintenance or course update. 

Determine Supplies Required. Given the number of students^ or number 
of student positions, calculate supplies required considering the ^ 
supplies per student ratio or the supplies per student position ratio, 
whichever is appropriate. 

Determine Miscellaneous Required, .Given the number of students or 
number of student positions, calculate miscellaneous required con- 
sidering the miscel 1 anepus per student ratio or the miscellaneous per 
student position ratio, whichever js appropriate. 

Determine Training Costs. Training costs are calculated by taking 
each of the preceding technical requirements times its unit cost. 

Student costs consider salary costs per student ye?r (AOB), student 
travel to and from school, as well as any travel while in training. 

Personnel costs consider instructor salaries and administrative 
salaries. 

New equipment costs consider the individual equipment purchase 
cost and annual maintenance cost. Based upon the defined deprecia- 
tion rate, the life of the equipment, and the length of the planning 
period, equipment may have a remaining value which will be sub- 
tracted from costs. (The .assumption here is that the equipment can 
be sold or put to an alternative use which is equal to its remaining 
value.) Used equipment valuation should be based upon its opportunity 
cost, that is, its value in alternative uses. 
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Facilities costs consider the total square footage requirements 
times a cost per square foot factor. 

Instructional materials costs consider the total development and 
maintenance time requirements and the cost per unit of time. 
Instructional materials, like equipment, may have some remaining 
value based upon some alternate use at the end of the planning 
period. .Considering a percentage factor for end of life value, 
the offset against total cost can be calculated to arrive at a 
net cost of instructional materials. 

Supplies and miscellaneous costs can be calculated from the ynit 
cost factors. 



© Aggregate Costs. Sum the costs on an annual basis over the planning 
period. A total (nondiscounted) cost is Obtained by summing the 
annual requirements. A more realistic economic value -which considers 
the time value of money is an overall value which has been discounted 
at a rate representing the annual cost of capital. The annual dis- 
counting should be performed and the discounted annual costs summed 
to obtain a priesent value cost figure for the program. 

® Develop Unit Factors. It is often valuable for comparative purposes 
to calculate the program cost per graduate, cost per studenf^rrp'ut , 
cost per student position, or cost per student day of training. This 
can be done with both the nondiscounted or discounted factors and 
displayed on an annual and total program basis. 

Job Cost Model. In general, the determination of seemingly quantifiable 
job effects will start with experts, knowledgeable of a specific job (or 
task) situation, predicting training variations or modifications upon the 
appropriate job variable; e.g., performance time, error rates, support 
requirements, etc. For life cycle planning purposes, generally done well 
in advance of implementation, savings of personnel will reflect in a re- 
duction of a number of costs including base pay, hazardous duty pay, 
transportation costs, retirement benefits, etc. Navy Military Manpower 
Billet Cost Data for Life Cycle Planning Purposes (NAVPERS 15163) provides 
costing information which can be applied once an estimate of the number 
of personnel affected has been made. Again, the difficulty is in evaluating 
the opportunity cost for time saved at a task or duty level. The question 
as to whether time saved can be put to some alternative use with value 
approximating the factors in the Billet Cost Data is more easily answered 
when preparing manning level estimates well in advance of implementation 
than it is when performance times on existing jobs are being affected by 
an innovation. 
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Some work has been accomplished in estimating the cost of on-the-job train- 
ing which could be one of the more frequently impacted variables in the 
assessment of training, innovations . While a generaT methodology has been 
developed for making OJT cost assessments , models would have to be 
developed for each job type (rating, etc.) potentially impacted. The 
elements of cost identified in an Air Force Study^ are: 

» Trainee time 

0 Instructor time , 

9 Delayed entry time 

9 Records management ti.ne 

9 Remedial training time 

0 Equipment and material. 

Various models for estimating costs in each category were examined and an 
appropriate one selected for predicting each cost element. If specific 
models and data such as those discussed above are available and appropriate, 
they should be used, however^ a general procedural approach for combining 
the data input factors specified in 06.02 is as follows. 

Q Determine Personnel Costs. Given the number of annual incumbents 

affected, salary rates can be identified based upon job classification, 
years service, and time-in-grade factors. The operational time utilized 
and unutilized time. cost become the basis for valuing the time saved or 
additional time used as a result of the proposed innovation. Overhead 
9 or support personnel can be determined from the support personnel ratio 
and total cost determined using the salary rate. Depending upon the 
nature of the innovation's impact over the planning period, total 
personnel costs could include the additional costs of recruiting/induc- 
tion, school training, and transportation. 

Q Determine Facilities Costs. Job support space required will be identi- 
fied and cost determined using the cost/square foot factor. 

Q Determine Job Performance Support Costs. Certain jobs require special- 
ized tools or equipment which can be costed using unit cost factors for 
each. 

*Samers , B.N. , Dunham, A.D. and Nor^hauser, P., 1974. The Development of a 
Methodology for Estimating the Cost of Air Force On-The-Job Training. Report 
No. AFHRL-TR-74-34, Contract F41609~72-C-0048, AFHRU, Brooks AFB, Texas, 
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• Determine Performance Change Costs. Some innovations may be modifying 
the rate of operational errors made by the job incumbent. The error 
rate and cost per error factors are intended to permit a cost estimate 
of this job attribute. 

• Aggregate Costs. Sum the costs on an annual basis over the planning 
period. A total (npndiscounted) cost is obtained by summing the 
annual requirements.'^ A more realistic economic value which considers 
the time value of money is an overall value which has been discounted 

•at a rate representing the annual cost of capital. The annual dis- 
counting should be performed and the discounted annual costs summed 
to obtain a present value cost figure for the program. 

• Develop Unit Factors. Comparisons of alternatives and before/after 
situations can be made if certain types of unit factors are developed 
such as, cost per job hour, cost per job hour utilized, error' cost 
per incumbent, support costs per incumbent, etc. 

4. Equipment Cost Model. 'Equipment costs are derived by combining the factors 
in the following general manner. 

• Determine Numbers of Equipment Affected. When an innovation causes 
equipment to be either displaced or procured, the numbers, by type, 
will be specified from Task 5. 

• Identify Related Resource Impacts. The displacement or procurement of 
equipment may affect other costable items such as: 

Refurbishing of facilities costs 

Transportation ^ 

Inventories (e.g., spare parts) 
^ - Maintenance 

Working funds 

Supplies 
' - Utilities. 



CP 
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Identify Unit Costs. For equipments and other eostable items, determine 
the unit costs from data bases or other sources. 

Equipment Purchase Costs. Equipment life (actual, not accounting) must 
be considered to determine the number of procurements to be made over 
the planning cycle as well as the potential cash flows resulting from 
an end-of-life equipment value. Purchase and depreciation policies 
will affect investment and other cash flows. 
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9 Equipment Utilization. Utilization would be an important considera- 
tion in arriving at an alternative use cost when displacing or using 
existing equipment. When equipment is already owned, its economic 
costs are approximated by estimating its value in alternative uses. 

Develop the incremental cost/savings value for each of the model cases 
run. This will be the difference between the baseline or existing system 
and the proposed systems for each decision tree outcome, A through D. If 
the innovation requires use of more than one model > the incremental values 
of each should be summed and assigned to the respective outeome. Be sure 
that paths {Z) , etc., contain the implementation costs identified in 04.03 
(Define Implementation Projects and Costs). These data will be used in 
Subtask 06.06 (Develop Decision Tree Assessment) and in Task 7 (Perform 
Financial Analysis). 
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TASK: PERFORM COST-BENEFITS ANALYSIS 
SUBTASK: DETERMINE BENEFIT/LIABILITY VALUES* 



DESCRIPTION ^' . 

The preceding subtasks have dealt with the tangible values of, costs and 
savings. Sometimes Intangible factors may have a significant effect upon 
the path a decision ultimately takes. The purpose of this sub task is to 
systematically develop value or utility for the otherwise nonquantifiable 
benefits and liabilities which result from application of the innovation. 
The value obtained from this procedure will be expressed in equivalent 
dollars and will be developed for each of the. decision tree outcomes ^o that 
the VcTlues can be added to the previously developed costs and/or savings. 

PROCEDURE 

1. Obtain the data assembled in 06.01 for the nonquantified benefits. and 
liabilities. Relevant variables may include process or procedural effec- 
tiven^s, attitudes, morale, communications, safety, security, environ- 
mental- quality, .manageability, controllability. Magnitude estimates and 
direction of change, relative importance, and confidence range will have 
been identified for each variable. 

2. Assess the relative effect of a single benefit/liability dimension for 
each decision tree outcome. Rank the outcomes according to their rela- 
tive value on a format as shown in Figure 6.4.1. Remember, only a single 
factor is' being considered at this time; for example, student morale. It 
is possible that this variable could be ranked in several ways depending 
upon the type of innovation. If the intended result of the innovation 
significantly improves student morale, then to accept, implement, and have 
user acceptance might be ranked first, while to accept, implement, and 
have user rejection (outcome B) might be ranked last. On the other hand, 
if student morale is negatively impacted (a liability of the proposed 
innovation), then to reject outright (outcome D) might be ranked first 
while to accept, implement, and have user acceptance (outcome A) might be 
ranked last. # 

3. Develop the weights for each of the ranked outcomes on a scale of 0 to 100. 
The first ranked outcome is assigned a value of 100 while the last ranked 
outcome is^ assigned a value of 0. Qetermine which outcome is neutral. By 
determining how much better or how much worse one outcome is in relation 

to another, each outcome can be assigned an intermediate value between 0 
and 100. Figure 6.4.2 shows a completed sheet where each outcome has been 
^ ^. ' 

*The approach outlined here was adapted from the "Handbook for Decision 
' Analysis", Decision and Designs, Inc., ONR Contract N00014-73C-0149, and 
4) should be reviewed prior to this phase of the assessment. 
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assigned a value. The rationale for the weights has also been included. 
The consistency of estimates can be checked by standard gambles. For 
example, if A = 100, B = 50, and D = 0, does B approximately equal a 
gamble between A and D each having a 50% probability of occurring? If 
this is a reasonable approximation, then the values are consistent. 

Select another vari able.^and repeat the analysis of steps 2 and 3. 



4 
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VALUE SCORE 



OUTCOME 
RANK LETTER 



OUTCOME NAME 



100 



90 



80 



70 



60 



— 50 



40 



30 



20 



10 
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FIGURE 6.4.1. BENEFIT OUTCOME ANALYSIS FORMAT 
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OUTCOME ANALYSIS BENEFIT RANKING AND VALUATION 
BENEFIT: STUDENT MORALE 



OUTCOME 
LETTER 



( 



OUTCOME NAME 



ACCEPT, IMPLEMENTc^ USER ACCEPT 



REJECT 



ACCEPT, FAIL TO IMPLEMENT 



ACCEPT, IMPLEMENT, USER REJECT 



RATIONALE: 

A CAUSES MOST FAVORABLE ) 
EFFECT ON STUDENT MORALE/ 

B CAUSES MOST ADVERSE EFFECT 

C IS ABOUT 25% AS UNFAVORABLE 
AS B, AND IS PLACED AT 60 

D IS A NEUTRAL OUTCOME AND 
IS PLACED AT 80 BECAUSE B IS 
FOUR TIMES AS UNFAVORABLE 
AS D IS FAVORABLE 



VALUE SCORE 



100 



— 90 



70 



50 



40 



30 



20 



■IGURE 6A2. BENEFIT OUTCOME ANALYSIS FORMAT (COMPLETED EXAMPLE) 
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TASK: PERFORM COST^NEFITS ANALYSIS 



SUBTASK: DEVELOP DECIS'KIN TREE ASSESSMENT 



DESCRIPTION 



The purpose of this subtask is to combine .the results of the cost model runs 
and the noncost benefits analysis by their relative importance weights to 
obtain a total value for each decision tree outcome in equivalent dollarsA 
The refined success probabilities and the eqjuivalent dol 1 ar val ues are \ 
entered into the decision tree assessment format and folded„back to find the 
decision path with the highest expected value. c. 



PROCEDURE 



Gather the results" of the cost model runs from 06.03, the benefit/ 
liabilities. value, scales from 06.04, the refined success probabilities 
from 06.05 (repeat of.04.02 and 04.04), and the importance values for ■ 
each variable assembled in 06.01. The data will be documented on a 
Benefit Outcome Summary Format as shown in Figure 6.6.1. 

Develop equivalent dollars for each relevant variable by weighting each 
variable'^.s scale by the relative importance factor. This procedure 
involves: ... 

a. Indicating the cost/savings value for each outcome. (See the example 
of the Training ACTUAL DOLLARS in Figure 6.6.2.) 

b. 'Determining the range of the scale for all outcomes. (In Figure 6.6.2 
ACTUAL DOLLARS go from +6.0 to -3.0, therefore, the range is 9.0.) 

c. Establishing the relative range of a new variable by the ratio of the 
new variable importance to the original dollar importance. (In 
Figure 6.6.2, the ratio of STUDENT MORALE to ACTUAL DOLLARS is 8:5 or 
1.33.) 

d. Apportioning the 0-100 scale, for the new variable obtained from the 
Benefit Outcome Analysis, across the new range. (In Figure 6.6.2, the 
new range is 1.33 X 9.0 or 12.0, therefore, the STUDENT MORALE variable 
in the 0-100 scale from Figure 6.4.2 is spread across this new range. 
Note that the range goes from +2.4 to -9.6 (12.0) because of the rela- 
tionship of the zero outcome (D) to the entire scale.) 
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3.0' Display the range .of values for all outcomes and for each variable on the 
Benefit Outcome Summary, am d summarize the total for each outcome as shown, 
on Figure 6*6. 3. ^ ; 

COMMENT: A major^ assumpti on in this summarization is that the utility 
attributes (e.g.; ACTUAL DOLLARS and STUDENT MORALE) can be added together.'^' 
Requirements (not discussed here) which include utility independence'^, pafrwise 
preferential independence and pairwise marginal ity, determine if the" additive 
form holds.* In general, it appears that most attributes which will be^ 
combined in the training domain will '^satisfy the requirements for additi vity."' 
Certain tes-ts can be made as outlined in the referencq,d literature if the * 
'assessor is- concerned about meeting* these requirements. 

4. Complete the decision tree assessment diagram by entering the refined 
success probabilities at the event nodes' and the values for each outcome 
at the end of each decision path. 

5. Fold back the decision tree as outlined in the procedure of 04.05.. Figure. 
6.6.4 provides an example of a completed decision tree. 

6* Review the decision variable results for reasonabi li ty . 



-^XOMMENT; The decision tree in the fonn presented does not include a considera- 
tion sequencing projects in a way that the decision to proceed is a function 
of, for example, successful completion of project A followed by successful 
completion of project B. If it is thought that this decision structure will 
exist and may have an effect upon the decision outcome , then the assessor may 
wafit to make a decision tree modification to reflect the decision situation 
njdr^e accurately. 

The process described above is only one facet of the decision process, the other 
is the financial analysis. However, if the decision Indicated as a result of 
this process does not appear reasonable from a global viewpoint, the individual 
decision components should be rechecked for reasonabill ty . ' ' 



*Wi 1 1 i am C. Giauque. 1975 Decision Making in the Naval Education and 
Training System. TAEG. 
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FIGURE 6.6.1. BENEFIT OUTCOME SUMMARY FORMAT 
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RELEVANT 
VARIABLE 



RELATIVE 
IMPORTANCE 



COMMENT 



ACTUAL 

DOLLARS 

(Training) 



6.0 2.0 1.0 0 



Cost/savings data 
from running Training 
Cost Model 



STUDENT 
MORALE 



c- 100 



60 80 Benefit outcome 
analysis format 
Figure 6.4.1 
Example 



(ADJUSTED)' 



1.6 -6.4 -1.6 



NOTE: The STUDENT MORALE variable is 8:6 or 1.33 times as important as the - 
ACTUAL DOLLARS, and outcome D is neutral or zero. Since the ACTUAL 
DOLLAR range is +6.0 to -0.0 or 6.0, and STUDENT MORALE is 1.33 times 
as important, the total STUDENT MORALE range is 1.33 X 6.0 = 8.0. 
Outcome A is 20% of the total range 1n the positive direction, there- 
fore, outcome A is 1.6. , Outcome B is 80% of the range in a negative 
direction, therefore, outcome B is -6.4. 



FIGURE 6.6.2. BENEFIT OUTCOME SUMiMARY 
(SAMPLE DATA) 
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OUTCOME ANALYSIS - BEWEFIT UTILITY ASSIGNMENT 
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FIGURE 6.6.3 BENEFIT OUTCOME SUIVllViARY (EXAMPLE) 
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TASK: PERFORM COST-BENEFITS ANALYSIS 
SUBTASK: PERFORM SENSITIVITY ANALYSIS 



ERIC 



DESCRIPTION 

The assessor may be faced with a decision in which there is very little 
difference in the decision variable values for the alternative choice; e g 
to accept without continued study versus to accept with continued study The 
purpose of this subtask is to present the assessor with an insight into the 
variables which might change the decision which is presently indicated This 
IS accomplished by determining which variables; e.g., estimates of success 
probabilities, relevant costable and noncostable variables, or measures of 
relative importance, may cause a change in the decision considering the con- 
fidence range that the estimator originally had in those particular variables 
Ihe results of this analysis may lead to the gathering of additional informa- 
tion or specific variables to which the decision is sensitive so that the" 
confidence in the estimate of their value can be improved. 

PROCEDURE 

1. Review the sensitivity analysis diagrams for the models used to develop 
the cost factors to determine if any of the variables affected by the 
proposed innovation are particularly sensitive. Figures 6.7.2 through 
6.7.7, are sample sensitivity graphs for the Training Cost Model. 

2. Identify the variables to be inputted to the sensitivity analysis program 
which will perform the following analytic steps. 

a. Determine if the inputted variables cause a change in the indicated' 
decision across the expressed confidence range.. 

b. If no confidence range, has been expressed; e.g., where the relative 
importance variable is concerned, the variable values for which the 
decision changes wil V be calculated. 

c. A sensitivity and probability that the indicated decision will change. 
. will be calculated for each variable that causes a change within the 

confidence range. 

^' The brej.k- even point value for each variable from b. and c. , above, 
will be calculated. 

e. An output report as shown in Figure 6.7.1 will be generated for the 
assessor to review. 
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COMMENT: The probability generated should be applied with caution since some 
major assumptions are involved in its calculation. First, as with the calcul 
tion of the mean or expected value, an approximation of the beta distribution 
IS assumed, therefore, the standard deviation is calculated by: 

S - £-£ 

6 

Where: S = standard deviation 

0 = optimistic estimate 

p = pessimistic estimate. 

A Z-value js calculated, assuming the beta approximates the normal distribu- 
tion by the formula: 

Z = b-VE 

^~ ■ 

Where: Z = Z-value 

b = break-even value 

VE= expected value (mean - see 06.01) 

S = standard deviation 

The cumulative probability is obtained from a table containing values of the 
standard normal distribution for various Z-vaTues. 

3. Review those variables to which the indicated decision appears sensitive 
and determine if additional information should be obtained to improve the 
confidence of the assessor that .the decision indicated is valid. 

4. If a "reject" decision is indicated and affirmed by the decision maker, 
create a tickler file entry as outlined in 04.06, otherwise proceed in 
the analysis to Task 7. 
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8 10 12 14 16 

PERCENT INCREASE IN VARIABLE 



FIGURE 6.7.2. TRAINING COST MODEL SENSITIVITY ANALYSIS 
AVERAGE DISCOUNTED COST PER GRADUATE 



■III-191 



TAEG REPORT NO. 12-3 




PERCENT INCREASE IN VARIABLE 




FIGURE 6.7.3. TRAINING COST MODEL SENSITIVITY ANALYSIS 
AVERAGE DISCOUNTED COST PER STUDENT POSITION 
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8 10 12 - 14 

PERCENT INCREASE IN VARIABLE 



FIGURE 6.7.4. TRAINING COSTTWODEL SENSITIVITY ANALYSIS 
AVERAGE DISCOUNTED CO^T PER GRADUATE 
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PERCENT INCREASE IN VARIABLE 



FIGURE 6.7.5, TRAINING COST MODEL SENSITIVITY ANALYSIS 
AVERAGE DISCOUNTED COST PER STUDENT POSITION 
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FIGURE 6.7.6. TRAINING COST MODEL SENSITIVITY ANALYSIS 
AVERAGE DISCOUNTED COST PER GRADUATE 
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TASK: PERFORM FINANCIAL- ANj\LYS IS . ^ 

Financial analysis adds an additional dimension to the data available for the 
decision maker to use' in reaching' a conclusion on the acceptance or rejection. 
The cost-benefits' analysis from Task 6, combined with the following financial 
analysis, in general, meets ^the requirements for an economic analysis. The 
definition of Economic Analysis as stated in SECNAVINST 7000.1 4A, dated 14 Mar 
1973 (Enclosure (1), page 2), is as follows: 

Economic Analysis : A systematic approach to the problem of choosing how to 
employ scarce resources and an investigation of the full implications of 
.achieving a given objective in the most efficient and effeoQ've manner. The ^ ' 
detfeVmination of efficiency and effectiveness is implicit in the assessment 
of the cost-effectiveness of alternative approaches and Is accomplished by: 

a. ^^stematicQlly identifying4;he benefits and' other outputs and . costs 
• associated with alternative programs, missions ,' and functions and/or 
of alternate ways for aQCorppJ-ishing a given program: (usually referred 
to as projects and activities). 

'b. Highlighting the sensitivity of a decision to the values of the key 
variables and assumptions on which decisions are based including 
technical, operational, schedule, and other, performance considerations. 

c. Evaluating, alternative methods of financing investments, such as lease 
'or buy; and ^ - 

d. Using benefits and costs to compare the relative merits of alterna- 
tives as an aid in : . 

^ 1) Making trade-offs between alternatives, 

2) ,Recommendihg the cost-effective alternative, and . 

3) In establishing or changing priorities. 

In terms of this definition, the entire ETAM can be considered an economic 
analysis. This task focuses on the quanti fiaf^le benefits (savings) and ^ 
liabilities (investment and recurring costs) to measure the ju'stifi ability of 
the proposed innovation in terms of its return on investment; and at the same 
time.it provides a. comparative analysis of several alternatives (identified in 
Task 2). The "General Guidelines for Doing Economic Analysis/Program Evalua- 
tion Studies" contained in SECNAVINST 7000. 14A should be reviewed in conjunction 
with this task, . * 
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SUBTASK \ . REFERENCE 

' !• Assemble ""the necessary input data to develop the required 07.01 ,01 
analyses, 

2. Develop Incremental cost/savings analysis for each 07.02,01 
alternative procedure to produce a Cost/Savings Source 

y. Report, 

3. Make analysis of each alternative in comparison with eaCh 07,03.01 
other alternative to produce an Alternatives Analysis 

-Report, I 

4. Perform sensitivity analysis of return on investment to 07,04,01 
selected variables, 

5. Create tickler file entry identifying criteria for 04,06,01 
reassessment of innovation if, at this point, the pnoposal 

does not appear feasible, ' ' 
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DESCRIPTION • . 

The assessor may or may not have carried each alternative generated in Task 2 
through the entire assessment process. If each alternative has been subjected 
yo a rigorous assessment, then the cost data.will bs available and can be 
introduced directly into the financial analysis. Othenvi^e a detailed cost 
analysis will have to be performed. .The data from the proposed innovation 
which have been carried through the assessment process to this point will, 
however, be available. 

.. 

PROCEDURE - • 

1. Obtain the cost data for the proposed innovation generated by the cost 
model run in 06.03. The cost data should reflect the results of Task 6 
which should have led to selection of a particular decision tree path. 
For example, if the particular path chosen includecl" investment in addi- 
tional studies or R&D projects, these costs should be included as inputs 
to the cost model . ■ 

2. Obtain the cost data for the baseline system. These data will have been 
^ ' used in Task 6 to calculate' the incremental costs/savings resulting from 

the proposed innovation. 

3. Obtain data available for any alternatives to the proposed innovation 
which may have been generated in Task 2. 
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TASK: PERFORM FINANCIAL ANALYSIS 
SUBTASK: DEVELOP COST/SAVINGS ANALYSIS 



DESCRIPTION 

In this subtask, each of the alternatives will be subjected to an incremental 
analysis across the entire range of application. It is assumed that the 
alternative approaches were designed to meet the same objective as outlined 
in Task 1 for the.^roposed innovation; therefore , .the range-of-effect 
analysis should lead to the same results. If, however, thi^s is not the case 
then a new range-of-effect analysis must be performed prior to costing a 
particular alternative. 

PROCEDURE 

1. Determine requirements for additional cost model runs. These may involve 
costing of alternatives as well asj^the costing of the speci'fic approach 
selected for the proposed innovation as a result of the assessment 
process in Task 6. 

2. Ensure that alternatives have been subjected to the appropriate range-of- 
effect analysis so that the entities to be costed (courses, jobs, equip- 
ments, etc.) are valid targets for appl icat i.^n of each alternative 
proposal. • ^ 

3. Establish the'input parameters and prepare input data sheets.as called 
out in 06.03. Each alternative will be incrementally assessed against 
every other alternative, the proposed innovation, and the baseline (or 
existing) system. A model nun input is prepared, therefore, for each of 
these cases. Two input factors b1j concern at this point are the infla- 
tion r^ite and the discount rate. The inflation rate permits an expression 
of annual ^figures in curt^ent dollars; i.e., the dollar values that would 
most likely be reflected in the budgets for those outyears where prices 
have increased. ^ The discount rate permits the time value of money to be 
[considered withi'n the^, overall decision process. Tho Department of^ 
Defense and Secretary of tne Navy have issued guidu.ice on the appropriate 
discount rate to be used. The present ratestated in DOD INSTRUCTION 
7041.3 and in SECNAV INSTRUCTION 7000. 14A is 10?^. This rate can be 
modified if \he analyst feels another rate is more appropriate. In the 
.private sector, this rate is sometimes adjusted to reflect risk. For 
example, low risk investments such as in replacement of machinery may use 
a discount rate as low as the current going-rate of interest. High risk 
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investments such as for R&D or new market-entry products may use a dis- 
count rate that is much higher; i.e., 25°^ or more. If an alternate rate 
IS selected, model runs should be made at both the alternate rate and the 
current DOD approved rate; i.e., 10%. 

4. Complete model runs for each of the cases prepared in step 3, and generate 
a Cost/Savings Source Report. A sample of this report is shown in 
Figure 7.2.1. 

COMMENT: The Cost/Savings Source Report shown in Figure 7.2.1 is one of three 
App'Ticati'Sns discussed in Section IV of this report under Computer 

The values of costs and savings represent the incremental change nr 
difference in the comparison of the effects of one alternative against 
the effects of another alternative. 

^" SAMif^^Sfx"'??^" Pi^cssnt value (NONDISCOUNTED TOTAL $ and PRESENT 
VALUh-LONbT ?) for each comparative incremental analysis. If these' 
figures are not positive, then the incremental rate of return for the 
alternati.ye in^ question will not be to the level expected; therefore, it 
may be desirable to exclude this alternative from further analysis. The 
information regarding any alternatives rejected at this, point should, how- 
ever, be carried forward for the review of the decision maker in Task 8 
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TASK: PERFORM FINANCIAL ANALYSIS 



SUBTASK: DEVLLOP ALTERNATIVES COMPARISON ANALYSIS 



DESCRIPTION 

The purpose of this subtask is to organize the data from Subtask 07.02 in such 
a way that alternatives (including the proposed innovation) which have been 
deemed reasonable can be compared with one another. The decision maker will 
have available certain financial measurements which are indications of the 
relative value of the alternatives, under consideration. 

- PROCEDURE 

.1. Select the analytic results from Subtask 07.02 for those alternatives 
accepted for further consideration. 

2. ^ Organize the data so that the incremental annual cost^ and savings are 

separated into the following categories: 

0 R&D costs - while these costs are not specifically displayed as an 
output of this subtask, they should be separately identified for 
subsequent funding requests should the proposal be accepted. 

0 Investment costs - these will be primarily equipment purchase casts; ' 
however, .other costs may be considered investment, such as start-up 
costs or rehabilitation and modification costs. The assessor should 
refer to SECNAV INSTRUCTION 7000. 14A for a more complete identifica- 
tion of the types of costs which might fall into this category. . 

t Recurring costs - these, are costs expected to recur on an annual basis, 
such as costs for personnel, maintenance of equipment, maintenance of 
instructional materials, supplies, etc. 

COMMENT: Data should have been classified according to the preceding cate- 
gories when originally introduced. into the costing cycle in Subtasks 06.02 
and 06.03. No major reorganization should be required at this time. 

3. Generate an Alternatives Analysis Report as presented in Figure 7.3.1. 
This report summarizes the incremental investment and recurring costs, and 
the savings for up to three alternatives, one of which may be the baseline 
or e.xisting system. The financial measurements calculated and displayed 
as a part of the Alternatives Analysis Report are: 

• Costs and savings are presented both in nondiscounted constant dollars 
and their present value, as obtained by discounting at the discount 
rate (percent) indicated. 
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07 



03 



02 



t • A savings Investment ratio is calculated as the ratio of the PV net 
recurring cst(-)/sav(+y to the total PV of new investment(-) • 

t The rate of return on investment is the rate which, if substituted 
for the discount rate used, would have made the total PV of cst(-)/ 
sav(+) equal to zero; i.e., the present value of the savings would 
have equaled the present value of the investment. 

$ The uniform annual cst('-)/sav(+) is the average cost of savings spread 
over the economic life of the alternative used for the analysis. It 
becomes .a useful number for comparing alternatives which have varying 
economic lives. 

COMMENT: The computational aspects of this report and its relationship to the 
Cost/Savings Source Report will be discussed in further detail in Section IV 
of this report under Computer Applications. 

4. Review the alternatives being compared, for those having favorable return 
on investment. If the proposed innovation shows the highest rate of 
return, then there -should be sufficient basis for its acceptance. How- 
ever, the following should be considered where the proposed innovation 
does not show the highest rate of return when compared with either the 
baseline (or existing) system or with one of the alternatives. 

• The amount of total investment required to accomplish each of the 
alternatives may be a critical factor in the ultimate decision to 
arcept one or the other. In Tasks 3 and 4, the funding as a possible 
constraint was assessed in terms of the potential risk it imposed on 
the implementation and use of the proposed innovation. An alternative, 
even with a lower rate of return, may be a more favorable approach if 
the overall risk is significantly lower due to a lesser funding 
requirement. 

• One concern in financial analysis is with the rate of return obtained 
on the investment of incremental dollars. If the rate of return on 
the incremental investment, in going from one alternative to another, 
is less than could be obtained through some alternate form of invest- 
ment, then perhaps the additional expenditure should be rejected. If, 
however, the investment dollars could not be put to better use, perhaps 
the proposal with the higher investment requirements should be accepted, 
even though its rate of return is less than an alternative with a 
lesser investment requirement. 

§ If an alternative proposal looks more favorable on a financial basis, 



it should be subjected to the analytic steps outlined in Task 6 to 
determine if it might be rejected because of greater nonquanti fied 
liabilities or because there is a change in the probabilities that 
the alternative can be successfully implemented, or successfully applied, 
by the user. 
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Determine the alternative or alternatives to be considered in the ultimate 
decision- process an-d perform the- required sensitivity analysis outlined in 
Subtask 07.04. 

If no candidates have an acceptable level of financial return and there is 
no hard requirement for selection of one of them for "mission accomplish- 
ment' reasons, then a tickler file entry should be created as outlined in 
Subtask 04.06. 
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TASK: PERFORM FINANCIAL ANALYSIS 
SUBTASK: PERFORM SENSITIVITY ANALYSIS 

DESCRIPTION ' • • . 

The results of comparing alternatives may have shown several to be fairly 
close to each other in terms of their rate of return, or the selected candidate 
may have had a rate of return close to the required return as stated in the ' 
wIt??M In either of these cases, the assessor may want to know which 

variables are particularly sensitive, such that their variation may possibly 
change the ultimate decision. The purpose of this subtask (as with the pre- 
ceding sensitivity analysis in Subtask 06.07) is to provide the assessor with 
insight into the variables which could cause a change in the decision if they 
were to vary over a reasonably expected range. A need for improving the con- 
fidence in the estimate of a particular variable may be indicated by this 
analysis which in turn may suggest the gathering of additional information. 

PROCEDURE 

1. Review the sensitivity an^alysis diagrams for the models used to develop 
the cost factors to determine if any of the variables affected by the 
proposed innovation are particularly sensitive. Sample sensitivity 
graphs of the Training Cost Model were presented in Subtask 06.07, 

2. Identify the varidbies to be inputted to the sensitivity analysis program 
whicn will perform the following steps. k a 

a. Determine if the inputted variables cause a change in the decision 
variable beyond some expressed limit, first considerina the limits 
of the confidence range expressed for the variables. A confidence 
range should have been expressed for each of the variables considered 
relevant to the application of the proposed innovation. However in 
the absence of a confidence range, the variable value at which the 
limit of the decision variable is reac'-ed will be calculated. 

b. The break-even value, or point at which the decision variable limit 
IS reached, will be calculated for each variable in which this occurs 
in between the two confidence limits. 

c. A sensitivity and probability that the indicated decision will change, 
will be calculated for each variable that causes a change within the 
confidence range. 

d. A report similar to that shown in Subtask 06.07, Figure 5.7.1, will be 
generated for the assessor to review. 
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SMhS'nJS7 "^^'f^'^::^ ^"^ interpretation of the statistical data 'kntioned in 
bubtask.ue. 07 apply here, also, and they should be reviewed. 

5. Review those variables to which the indicated decision appears sensitive 
and determine if additional information should be obtained to improve the 
confidence of the assessor that the decision indicated is valid. 

^' generated in this task will be used in Task 8 as 

a part of the overall decision-making format. If, at this time, however, 
there appears to be an obvious basis for rejecting the proposed innovation 
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TASK: ' MAKE ACCEPT/REJECT/STUDY DECISION 



TASK SUBTASK PAGE 
08 00 02 



This task is to provide guidance to the assessor in orxi^r that a coherent 
Idecision-making format can be structured for presentation to the ultimate 
decision makers. It provides examples of both format and content which are 
, recommended for presentation. In addition, it addresses some of the questions 
the decision maker might have, in order to suggest back up detai'l that should 
also be made available as a supplement/to the presentation. The objective of 
ETAM has been to ensure thiit infortnat-yon has not been lost in the process of ' 
quantifying and summarizing the data. / Therefore, -the information available 
to the^^decision maker w-i 11 /be in consonance with this objective. 

SUBTAsfe_^ . I -y . REFERENCE 

1. Assenible the outputs from each step of the assessment '08.01.01 
^ procedure, /Tasks 1 through 7. 

2. Define, the/presentation format so that it is consistent 08.02.01, 
with the nequirements of the decision maker. 

•3. Prepare tfje material to be presented to the decision ^ 08.03.01 
maker. 

4. Review thk material planned for presentation in the 08.04.01 
context Qf potential questions which the decision 

• maker might have. . 

5. Obtain tile decision and define a plan for proceeding ' 08.05.01 
b^sed upon the decision. 
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TASK: MAKE ACCEPT/REJECT/STUDY DECISION 
SUBTASK: ASSEMBLE TASK OUTPUT DATA 



DESCRIPTION 

' * 

Assemble the descriptive and analytic data resulting from each of the ETAM 
tasks. These data should have been maintained in"a project file and should, 
therefore, be readily available. 

PROCEDURE ■ • 

1. Obtain the following data which will be used in preparing the presentation 
to the decision' maker. ■ ' 

a. Task 1 - all information describing. the proposed innovation and any ' 
of the alternatives subjected to the procedures in Task 1. Of primary 
interest are the objectives of the innovation and any assumptions made 
r with regard, to , the application of the innovation. 

tL_>Task 2 - any alternatives which were considered, whether they were 
subj-^cted to the entire assessment procedure or not. 

c. Task 3 - the results of the preliminary risk assessment identifying 
major stumbling blocks to the successful applic3tion'of the innovation. 

d. Task 4 - the specific projects defined in order to reduce identified 
risks, and the assessment of costs and impact upon risk reduction. 

e. Task 5 - the search parameters derived from evaluating the innovation 
in relation to a number of taxonomic structures the particular 
entities found as a result of the searches, and the benefits pattern ' 
attached to the successful application of the proposed innovation. 

" •{ 

f. Task 6 - the results of-each analytic step which led to the formulation 
» of the final decision tree structure, and the sensitivity analysis data 

related to the decision tree. 

g. Task 7.- the Cost/Savings Source Report for each of the alternatives ' 
under consideration, and the Alternatives Analysis Report comparing 
alternatives on a financial basis. Sensitivity analysis data relative 
to the selected alternatives should also be obtained. 

h. Any additional references, supporting rationale, or expert judgments 
obtained in the assessment process should be' obtained. 
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TASK: MAKE ACCEPT/REJECT/STUDY DECISION 
SUBTASK: DEFINE PRESENTATION FORMAT 



DESCRIPTION 

The decision maker will undoubtedly have a limited amount of time to review 
the proposal and the analytic results. Also, the decision maker will have 
format preferences which aid in the comparison of numerous" requests for 
resources. A third consideration is the level of technical understanding of 
the problem hnd of concepts inherent in the solution that the decision maker 
might have. The purpose of this subtask is to provide guidance to the 
assessor in arriving at a presentation format which meets the above needs. 

PROCEDURE 

1. Obtain information on the preferences of the decision maker for particular 
types t)f formats. These preferences should be in terms of medium and 
style. 

2. Determine the amount of time which will be available for the review and 
decision process and the conditions under which the review will be 
conducted. 

3. Determine the degree of background which the decision maker has in both 
the problem addressed/by the innovation, and in the various technical 
aspects of the solution. 

j. Define a format which considers the needs of the decision inaker, but at 
' the sarae^time ensures that the relative merits of the proposal can be 
adequately displayed. Some presentation techniques are outlined in a 
Department of Defense publication titled, "Economic Analysis Handbook. 
2nd Edition," 

RATIONALE " 

This final step of presenting the results of substantial analytic work may 
have a significant effect upon the decision outcome. Even a sound and justi- 
fied proposal may fail to gain acceptance because'of shortcomings in the 
presentation format. 
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TASK: MAKE ACCEPT/REJECT/STUDY DECISION 



SUBTASK: PREPARE MATERIAL FOR PRESENTATION 



DESCRIPTION " . 

Once a format has been defined for the presentation, the content of the 
analysis may have to be restructured to fit. It is important, though, that 
the final content be organized so that the decision maker is presented with 
a coherent view of the innovation, its objective in terms of application., the 
systematic analysis to which it was subjected, the role of experts, in making 
critical aspects of the analysis, and of the finaT^ conclusions and recommenda- 
tions resulting from the analysis. The purpose of this subtask is to provide 
the assessor with guidance in structuring the content for presentation within 
the identified format. 

PROCEDURE " * 

1. Organize the presentation sequence so that it represents a logical pro- 
gression from a statement of the problem, in tew,s that the decision 
maker will understand, to any background data necessary for understanding 
why the objective of the innovation is reasonable, to a statement of the 
objective, to the proposed approach which meets the stated objective, and 
finally to the various supporting data necessary for the decision.^ 

2. Prepare conclusions which are consistent with the supporting analytic data 
and recommendations which support the conclusions. 

3. Consider alternative levels of presentation which Will still retain the ' 
coherence intended. 

4. Prepare a concise summary which reiterates the general conclusions and 
recommendations and includes a basic supporting rationale for each. 
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TASK: MAKE ACCEPT/REJECT/STUDY DECISION 



SUBTASK: REVIEW PRESENTATION MATERIAL 



DESCRIPTION 

t 

The purpose of this subtask is to suggest possible questions that the decision 
maker might have with regard to the appli-cation of the innovations^ the analytic 
process followed and the data inputs to it, and conclusions and recommendations 
It is assumed that the assessment process has considered all of these potential 
concerns of the decision maker at one stage of the procedure or'another. There 
fore, the intent here is not so much to ensure that they have been adequately 
addressed, but rather to serve as a' guide to the presenter in preparing for 
the presentation. 

PROCEDURE 

1. Consider the following questions in reviewing the material to be presented 
to the decision maker.* 

^ a. Objective, assumptions, and alternatives. 

Is the problem stated the real .problem? 

Are all reasonable assumptions identified and explained? . 

Are assumptions too restrictive? Too broad? 

Are intuitive judgments identified as such? ,^Are uncertainties treated 
as facts? Can the facts be verified? 
Are any feasible alternatives omitted? 

Are the alternatives well-defined and discrete? Do they overlap? 

b. Benefit/liability determination. 

Were all relevant benefits and liabilities identified? 

Was the appropriate level of relative importance assigned to each 

benefit/liability? 

Were the criteria used to measure benefit/liability justified by the 
context of the study? 

Was the benefit/liability, in fact, unmeasurable? Has there been a 

rational assessment of nonquantifiabl<e factors? 

Was expert opinion used? Were these .experts properly qualified? 



^Adapted from the "Economic Analysis Handbook, 2nd Edition," prepared by the 
Depa^rtment of Defense. 
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c. Risk assessment. 

Were all potential ris1< ^factors considered? 

Was the appropriate level of risk attached to each? 

Are R&D or study projects to address risk factors reasonable? 

Was the reassessment of risk given these projects reasonable? 

d. Cost estimates . 

What costing method was used? It is appropriate? 

Are all relevant costs included? Are overhead costs appropriately 

included or excluded? 

Is the rationale for including or excluding certain costs included? 
Are the opportunity costs properly identified? Are aunk costs 
excluded? 

Are the sources of cost data included? Are they accurate? 

Are the cost estimating relationships valid, if the parametric method 

was used? Are extrapolations used without proof? 

Was an appropriate discount rate used for obtaining present value? 



Prepare any supplementary material suggested by the potential questions 
which the decision maker might have. 
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TASK: MAKE ACCEPT/REJECT/STUDY DECISION 



SUBTASK: OBTAIN DECISION AND DEFINE PLAN 



DESCRIPTION 

The preceding steps should lead to either a firm or a tentative decision upon 
which some degree of follow-on activity can be based. This subtask suggests 
some of the follow-on activities to be pursued based upon different decision 
outcomes. 

PROCEDURE 

1. Obtain a decision (firm or tentati ve) 'to accept, continue to stucly, or 
reject the proposed innovation or an alternative. 

2. Perform the following activities based upon the decision outcome. 

a. Decision to accept. 

Prepare a detailed implementation with supporting rationale for each 

phase of development, implementation, and testing. 

Identify resource needs. 

Identify checkpoints and major milestones. 

Develop auditing procedures. 

b. Decision to accept with continued study. 

This decision requires essentially the same items mentioned above, 
except that R&D^ and study plans may have to be developed, and go no~go 
criteria established if appropriate to the intent of the R&D or study 
projects. 

c. Decision to reject. 

Docuijient rationale for rejection. 

Estimate potential for and timing of future application. 
Create tickler file entry per Subtask 04.06. 
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SECTION IV 
COMPUTER APPLICATIONS 



The Educational Technology Assessment Model is an organized grouping of proce- 
dural sequences jfchat provide a structure for efficiently and effectively 
assessing the true value of a proposed innovation in training. Efficiency is, 
however» highly dependent upon the assessor's ability to access data when 
required, rapidly make calculations using selectable routines (e.g.. Training 
Cost Model), modify input data and make recalculations, and obtain "output 
reports with suitable content and format for effective analysis and decision 
making. These requirements for an efficient assessment processing structure 
suggest the need for: 

1. Highly organized data bases with rapid search and retrieval capability. 

2. A collection of automated routines which can be selected at specific 
stages of the assessment process for making calculations or manipulating 
data. 

3. A project mainten'ance facility which will serve as a repository for: 



a. 


Descriptive data on the innovation. 


b. 


Input data factors. 


c. 


Results of data base searches. 


d. 


Intermediate calculated results. 


e. 


Implementation subproject descriptive and cost data. 


•f 
1 . 


Final analytic results. 



4. Ar. "Executive Control System" capable of sequencing data retrieval, data 
manipulation, calculations, interactive requirements, and output genera- 
tion based upon the analytic needs at a particular phase of the assess- 
ment. 

The purpose of this section is to describe the general framework of an auto- 
mated data system capable of fulfilling these needs, and to provide develop- 
ment guidance in the form of design specifications for those who might under- 
take- the ^modeling and programming of such a system. 

ETAM AUTOMATED DATA SYSTEM STRUCTURE ^ ' . . 

The automated data system is composed of five major programs which control the 
flow of data and sequence of calculations, as well as provide input data paths 
to the data bases. The interrelationships between these five programs and the 
data bases are shown in Figure IV-K Following are brief descriptions of each 
of the programs. 
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DESCRIPTION STRUCTURING PROGRAM. Source descriptive and project data are 
entered via the. Description Structuring Program which performs a missing data 
edit and identifies either the need for additional data or the default data 
pattern. Project identifications are established to ensure traceability 
throughout the entire analytic and data transfer process. Overall data struc- 
turing, as it relates to the organization of the project file data base, is 
performed here. Any batch data input (e.g., cost model input parameters) is 
entered through the Description Structuring Program. 

RANGE-OF-EFFECT PROGRAM;. The purpose of this program is to formulate the pro- 
cessing pattern which is inputted to the Assessment Program. It will provide 
the user with preliminary guidance in preparing the taxonomic descriptors 
which will.be used to search out the innovation's full range of application. 
Once the search characteristics have been defined, either interactively or off- 
line, the specific se'arch parameter for the equipment, course, or job data 
bases are generated. The Range-of-Effect Program also structures and edits the 
benefits pattern to be assigned specific attributes of entities selected from 
the data bases; e.g., magnitude, importance, and confidence in the attrition 
factor for course 12AB. Specific allocation rules, which provide further 
selectivity to the final Assessment Program input, are fonnulated; e.g., 
applies only to attritions identified with a prior setback! Some Innovation 
benefits will be direcly costable, which may preclude. the need for allocation 
rules or benefits patterns, other than the^ direct amount of dollars displaced. 
The project fije data base, containing abbreviated records for all Navy courses, 
jobs, etc., and user interactive inputs are the primary sources of data upon 
which the Range-of-Effect Program operates. Any of the entities extracted 
through a data search operation will be passed to the^ extracted data base for 
further manipulation. 

ASSESSMENT PROGRAM. The Assessment Program perforfiis' major computational steps 
in the analytic process. It contains four basic types of computational capa- 
bilities, namely, cost models, decisi^on tree evaluators, financial analysis rou- 
tines, and sensitivity analysis routines. In conjunction' with these computa- 
tional routines, basic input editing and default data selection functions are 
carried out within the Assessment Program. These routines will communicate 
with the user to identify additional data requirements, or to point out^the 
characteristics of particular default data being used. Interim, as well as. 
final results, will be retained in the ETAM Working Data Base for additional 
processing or modification, if desired. An aggregation routine will provide ' 
the means of summarizing the data from multiple selected entities (e.g., jobs), 
so that the computational routines can be run on aggregated data. 

USER' INTERACTIVE PROGRAM. This program is effectively the "Executive Control 
System" for the entire system. It includes a skeleton monitor program which 
controls the sequence of operation for the various routines identified in the 
other major programs. The User Interactive Program also contains input-output 
routines for communicating with the user through a terminal and graphic display, 
or through a set of standard, selectable reports, produced as output from the 
various routines in the major programs. In certain routines, such as the deci- 
sion tree evaluation, within the assessment program, interaction with the user 
may be highly desirable in formulating parameters or in generating the scenarios 
required for each decision tree outcome. The routines to accomplish this inter- 
action are a part of the User Interactive Program. 
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DATA BASE MAINTENANCE PROGRAM. This 1s a program containing a number of stan- 
dard routines required to maintain the Master Data Bases. They provide the 
facilities for initial file loading, file updating, file editing, and for 
unloading or dumping the file to some output device. A similar set .of main- 
tenance routines will also be required for the ETAM Working Data Base. 

DATA BASE COf^S I DERATIONS 

The general detail relating to the structure of the ETAM data bases is covered 
in the portion of this section titled "ETAM PROGRAM DESIGN SPECIFICATION." The 
discussion here is limited to data base considerations that apply only to the 
external use of information in the traintng enterprise for retrieval and ana- 
lytic purposes. That is, the major concern here 1s with the disciplines for 
the naming of classes of data as users will interact with those names and the 
data. The internal logic and coding of data structures will not be discussed. 

PURPOSE. The specific purposes for examining data base factors here are 
directed towards the issues of matching the characterization of a proposed 
innovation or change in training practice or technology, and the characteriza- 
tions of what are or may be affected by that potential change. Thus, a nomen- 
clature is required for the classification of the various kinds and patterns of 
benefits and liabilities that are associated with any aspect of the instruction 
al process. If the nomenclature is to serve additional practical purposes for 
evaluation, the variables and entities that are identified should be associated 
with cost factors. Thus> a given benefit factor can be directly associated 
with cost because it is also a cost factor, and a term in the classification 
scheme can be used, both to access data about benefits accruing from that term, 
and also data about the costs associated with that term. The value of this 
capability in assessing the desirability of a proposed change in method or 
resource used should be clear. 

But the classification scheme should not only identify individual benefits and 
patterns of benefits, and costs associated with those benefits for the purpose 
of a technique for evaluating proposed changes in Instructional technology, it 
should also represent the conceptual structure, or model, of instructional 
technology in its broadest sense. In this sense, the data base classification 
scheme becomes a taxonomy. 

The treatment here is not a rationale for a taxonomic structure in educational 
technology, nor will it contain justification for the nomenclature proposed 
hiere. It will not represent all of the cost variables of the training enter- 
prise by any means,— rather it will identify the major functional variables 
in education and training around which, for some purposes, cost data should be 
clustered. These purposes are clustered in the potential manifestations of 
changes in educational technology. 

Specifically, the purpose of the nomenclatures to be proposed in the following 
paragraphs is to provide key terms to apply to the description of instructional 
vehicles and media, to students and job incumbents, to jobs and job tasks, and 
to courses and constituent course units and objectives. It is not proposed 
that only those descriptors or index terms proposed will be sufficient for all 
purposes in the use^of the training system's data bases. 
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The Data Sets . A data s^t may be thought of as a tabular arrangement of data 
In which a primary list of entities, such as student name, Is associated with 
a set of attribute names ^ such as '"date of enlistment," "training courses taken," 
"job. skills," and so forth. An entry in the table under an attribute may be ^ 
quantitative, qualitative, or text description. The key factor in the practical 
use of a data set is in the name of the primary entities and the names of the 
attributes that make up the headings in the table. The name of a data set is 
generally based on the name of the primary entity. 

The benefits and liabilities analysis in ETAM calls for four major data sets. 
These are: 

4? 

Instructional vehicles and their functional attributes 
Courses and their instructional attributes 

Jobs and job tasks and their functional attributes . 

Student-s and their learning and performance attributes. 

The search arguments developed by the analytic aspects of the assessment pro- 
cess will be directed towards the identification and retrieval of sets of 
entities (and entity descriptions and contexts) potentially relevant to the. 
applicability of the innovation or proposed change. Task 5 in the ETAM 
procedures identifies and defines' all of the following terms and Implicitly 
develops their practical justification. 

Instructional Vehicles: Types and Functional Attributes . The characterizations 
are based on vehicle type, vehicle properties, and on Tnstructional context. ^ 

Vehicle Type 

Instructor 

Static graphics ' 
Animated graphics 
Static physical models 
Procedural trainers 

Symbolic 

Physical mockup 
Task simulators. 

This typology is not intended to be exhaustive, but to represent a minimum set 
of useful distinctions. Each type will have subset categories, and any given . 
device or vehicle may overlap more than one type. Secondary characterizations 
may, of course, be added to the description of any entity. 
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A second form of characterization of vehicles 1s- by the way 1n which they manl 
fest or implement each of a set of properties. Properties of instructional 
vehicles may be differentiated into those primarily directed towards instruc- 
tlonaT functions and those primarily directed towards' util izati on functions. 

The following instructional functions are exhaustive of the variables that 
comprise any capability for learning. 

Instructional Functions * 

Type of stimulus presented (e.g., visual, auditory, kinesthetic) ' 

\ Type of content presented (e.g., static, dynamic, dlagranmatlc) 

lype of presentational sequence 

Selection source for sequencing 

Type\^ external control operated by student 

Feedbac^\presentat1on logic 

Response e^^uatfon logic. 

A second class of properties associated with Instructional devices Is that of 
utilization functions. Nd list of utilization functions can be complete and 
exhaustive. A salection can be made, however, of these functions which, taken 
together as a set, make up those generally most important from a practical 
sense. 

Utilization Functions 

Programmabili ty 

Portability 

Multiple usage 

Operating simplification or usage 
Maintenance 

Updatability in instructional function^ 
Combining of instructional capabilities 
Modularization of training content 

Translatability of a training requirement into a training content. 

Another characterization of instructional vehicles is not an inherent property, 
but is based on the class of use for which it is applied or may be intended. 
In the specific sense, every item of a given instructional vehicle should be 
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..cL''^''^^''J'"'"^ ^"'^ specific instructional objective in' which 

V h clP PnH ° cross-reference can be ma.de between the instructional 

charactPH^ Hnn nf J' u""*'"* 'Addition, a general 

m.rhp ncif ? °^ training objective for which it is generally used or useful 
may be useful for inquiry and search. uberui 

Class of Training Objective 
Reference knowledge 

Knowledge: task-specif i c and enabl ing 
Task skill format 
Skill training 

Procedural , ' 

Tracking. 

A final category of classification for the instructional entity is "Where Used" 

iSTn^Mp-.J*^'' '^t'^'l'' '''''' objectives and ojfler data 

that enable the searcher to link the device to its specific applications, which 
n turn may identify quantifications of how much the device is used what is 

lorZl n/Z°' '''' "^'y so on These a - 

gones of data are more likely to be associated with course data sets if the 
latter keeps data about training operations. ■ 

Jobs and Job Tasks. The primary objective of the training enterprise is to 
take training requirements, bAsed on skill requirements, and convert them into 
instruction which converts input student aptitudes and skill levels into on- 
the-job competences. Competence in jobs and taSks is the ultimate criterion 
of training benefits. It is. therefore, the dependent variable to wh ch a?" 
otner variables contribute and, therefore, should reference. 

The characterization of job tasks 'should erfable the categories to be linked to 

o'p'ations Ea^i h'\' cate^gories,. and to instructional vehicles nd 

-operations. Each job task categorization should not only assist in formulatina 
a search argument, but also differentiate a type of learning objective aSd/or 
an instructional process or resource. The category structure must 0Dtimi7P 
between two confl i cting criteria: ease and simg?^city of e rn no anT se vs ^ 
precision within practical limits imposed by-^liriF^dure and ^he obje • 

following structure is offered. Type of 

ra h r Ln snpri^ir'''?;°" 'fn'^"'' l^^ ^''/^ contextual 
ratner than specific. It would assist in identifying a duty within a job. 

Type of Skill 

Tracking , 
Procedural 
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S 



Decision-Making 
Constructing 

Leadership/Interpersonal . 



ch!rJctPH7! I^^^c'"* ' of ^descriptors for jobs and tasks are those that 
S task finctinn. TZl'^' characterize task structure elements 

??oV?ne' llt.T'ti: ll^tr'' '""'''^'r''^ °^ trans^erabfJftrSr^k'?!-]" 



Task Support Characteristics 
Equipment, objects used 
Environments 
Tools, 

Reference Information 

Operations (task elements) 

Criteria of performance: Time, errors. 



A supplemental characterization of tasks is in the relative weiqhtinas and 
contexts of performance components., or of elements or function thaE^comSrise 
the structure of an^ task. Each of these functions, as identified be W has 
somewhat different instructional requirements, but .the instruct onal require- 
ments for the given task function are about the same wherever thaStasT 

NSJ?Stu?I'''£f?r;ho^^"'* one can effectively teach "Identification and 
Nomenclature with the same instructional setup, whatever the job or duty may 



Task Structure Elements 

Goal projection ^ • 

Scan/detect 

Identify 

Interpret \ 
Procedure operations . 
Track 
Decl de 

Construct/plan 
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Motor performance • 
Interpersonal interaction 
Recall task-cycle information 
Recall enabling information. 



fvery-task^ has some representation or weighting of all of these -components. But 
• vfrom the sirarvdpoint of skill learning, each task has a somewhat different mix of 

these components, and most tasks tend to have high weightings on one or two. In^ 
.fact, the preceding structure suggests how tasks may be identified and differen- 
•tiated from each other for training and for evaluation of performance purposes. 

If job tasks are identified by preponderant Task Structure Elements, and also by 
descriptions under each. of the specified Task Support Characteristics, the 
nature of the skill is readily identified, transferability of s'kill from one job 
to another with respect to this task can be at least grossly estimated and the^ 
nature of the training appropriate to the task is iniplied. 

Furthermore, irtraining objectives for skill learning are similarly coded; it 
becomes easy to cross-reference a skill requirement, or an on-the-job competence, 
with the training objective. Not only would this facilitate the prediction of 
the effects of making changes in some aspect of instroictional technology, it ' 
would, also facilitate the development of an archival data base of experiences - 
with alternatives in instructional technique applied to various skill learning 
contexts.. If these archives cpu'ld be accessed by this taxonomy, ex post facto 
research could be conducted with historical data. The value of such data for 
•at least preliminary asse'sjsment of proposed innovations, in technology could be 
substantial. ' . , 

Student g-h aracteristics . The. Navy has developed extensive classificatory struc- 
tures fdr x^haracterl zing 'entrant personnel -hy tests, interviews, and observa- .- 
tions. VEcr purposes of this- project, these characterizations may remain as 
they are.- Of particular relevance are the patterns. of test,s",and scores that 
establish requirements for entrance to variou? curricula and courses in Navy 
training. The more demanding the requirement's fpr e'ntering and passing a 
course,, the higher the relative mar.jDower cost 'to the Navy. It is important ' 
that the data base on students, enable retrieval of t^j variables which are 
pertinent to entrance to instructional programs and courses, and to 'success, or 
the lack of it, in ttmse programs ana courises. • • ' 



Training Courses, cfiaracterizations of skills and' tasks, provide one basis for 
applying descriptors for courses aime,d at skill learniny. But there are also 
many courses directed towards knowledges intended to pRovide contextual back- 
ground to students. These instructional objectives tend to share' greater 
similarities in instructional vehici'cs and techniques than are common to skill 
1 earn ir^g, hence', the importance of making distinctions among subsets of such 
material is not so great. The following varieties of reference knowledge are 
by no means exhaustive ^/ reference knowledges, but are typi cal'ly-. represented 
in courses and in objectives within courses. They may not have the same names 
as are given. 



TAEG REPORT NO. 12-3 

i 

Type of Reference Knowledge 
System purposes " 
Organizational roles 
Contexts of operations 

♦ 

Theory of operation 

Organization rules, constraints . 

Nonwprk related content. 

Idealized instructional technique could develop methods of teaching theory of 
operations in ways very clearly different from instruction on .these pther topics. 
. A tenuous basis for the categories does exist. 

Another class of knowledge deserves differentiations. These are knowledges that 
enable the individual to learn the skill more efficiently, or to perform.it. 

Types o^Enabling Knowledge 

Operational goal criteria 

Nomenclature,- identifications, and locations of work objects 

-Procedure descriptions 

Facts applicable to job performance. 

Since enabling knowledges can be directly associated with skill' learning 
objectives, they can be evaluated against a practical criterion: their contri- 
bution to learning the skill. It is triereifore useful to ideniify enabling 
knowledges in course and course element descriptions. 

Skills of any kind are learned in a series of stages. A given course, or 
element or unit within a course, may apply to a given stage of learning. The 
content and Instructional technique and vehicle appropriate to one stage may 
be inappropriate \to another, and in highly cost-effective training systems, 
these distinctions are recognized as practical. It is, therefore, important in - 
the sense of being useful, to identify a course, and especially units within 
courses, not only by the kind of skill being taught but also by the skill level 
being learned.. The following structure-is used in ETAM. (As in all of the 
other classifications offered in this section^ -the reader can find definitions, 
examples, and rationales for every item in each list in the ETAM procedures. 
Section III of this report.) 

• Stages of Learning 

. Orientd'tion 

Nomenclature and identifications 
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Mediated procedures 
Mediated performance 
Automated performance 

Highly skilled q^erformance in work context 

Unusual task conditions - . . * 

Refresher learning 

Transfer of training to new. tasks, environments. 

The last item is not a stage but a condition applicable to any stage, and is a 
realistic condition to identify for the prograiimiing of training and other 
personnel design and decision situations. 

The empirical data that should be retained and accessible for any course unit 
or element that have been characterized by the various sets of descriptors 
specified here, are the following. 

Empirical Data to be Retained for Courses and Course Elements 

Aptitude level for acceptance to course: should include not only 
cutoff scores^but actual distributions of aptitude scores of students 
entering a given class. 

Training time per course and per course element or unit of instruction. 
Attrition rate per course and per course element orunit* 

Performance level achieved by student in a given class and in course 
elements within the course; mean scores and distribution of scores- 
Individual student performance scores should be relatable to aptitude 
scores, entering skill characteristics, scores on other courses. 

These empirical data provide targets against which innovations may be directed 
and evaluated. If feedback data on students can be obtained from on-the-job 
performance, the targeting can be more precise and more reliably relevant to . 
training mission objectives. 

CONCLUSIONS. It should be emphasized that a categorical structure, intended to \ 
serve practical search and retrieval activities with human interaction as part 
of the operation, does not require either mutual exclusivity of data categories 
nor exhaustiveness. These may be the desiderata of science. Although desirable 
in practical matters, they are not essential. This lesser requirement for 
practicality is indeed fortunate because, as R. B. Miller has shown on logical 
grounds elsewhere, it is not possible to define the meanings of "functions," 
"purposes, and "properties" in ways that are mutually exclusive and exhaustive. 
These criteria can be applied only to objects or to the logical definitions of 
sets of symbols treated only as synibols (i.e., as "objects"). 
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The relative ease or difficulty with which indexers .can learn and apply this 
taxonomic structure to instructional vehicles, jobs and tasks, and courses 
needs to be determined. At least some of the terminology, and many of the 
concepts, are already in practice, although not systematically. It also 
remains to be determined how easily researchers and others who wish to use the 
data base can learn and use this structured terminology. Indexing and search- 
ing are reciprocal activities; effort spent in indexing can be retrieved by 
ease of search. i 

ETAM PROGRAM DESIGN SPECIFI CATIONS 

This section sets down the functional specifications for each of the programs 
proposed for development within the overall ETAM automated data systems 
structure. System components exist at various levels of specification and 
design; some have been taken to a preliminary development stage. Some of the 
detail has been included here to provide examples of these various levels; 
however, not every program is carried to the depth of the examples. The 
intent is to provide sufficient guidance to a modeler and progranmer for 
development of the individual programs within an integrated system's framework 
It must be pointed out that a major portion of the design specifications is 
contained within the ETAM procedures in Section III of this report. Program 
developers must be thoroughly familiar with the overall ETAM structure prior to 
undertaking modeling and programming tasks. 

OVERALL SYSTEM OBJECTIVES. The user is a major concern in the development of 
any complex interactive automated data system. The following objectives are 
Intended as overall guidance to the development process to ensure a high level 
of user orientation is achieved. 

1. Operational Ease. 

There will be no requirement that the ultimate user be familiar with either 
the software or hardware design in order to solve his problems using the 
system. Tfiis implies an interface with the computet" in a language which 
is presently within the user's repertory, and sufficient prompting and 
assistance to promote simplicity in problem entry, as well as in output 
interpretation. 

2. Interactive Capability. 

Much of the data entry initially will be in batch form; however, once the 
basic analytic process has begun, a high degree of user interaction will 
be a part of the overall design structure. The user will be ab^e to 
specify solution formats, to modify and override existing data, to call 
upon specific calculation or data manipulation routines, or- to specify 
output report formats. The assistance provided the user through on-line 
prompting will be sufficient to permit rapid convergency toward problem 
solution and decision. 

3. Modular Design. 

As the user undertakes the assessment process, additional calculation and 
data manipulation requirements may be identified. The software system will 
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be designed in such a way that existing routine*; can be easily modified 
or deleted without impacting other components of the system, and so that 
additional routines can be developed and easily incorporated as a new 
module within the existing system framev/ork. This facility is necessary 
to maintain a "general purpose" character to the assessment process, so 
that analytic processes'unique to assessing certain types of innovations 
can.be added as they are identified. 

4. Data Base Maintenance. - . 

Several data bases have been identified to interface with the ETAM pro- 
grams. The intent is to access existing Navy data bases, insofar as is 
possible, so that existing data base maintenance approaches can be used. 
The objective is to minimize the amount of additional data to be collected 
and stored. However, to facilitate the EJM procedures an ETAM Working 
Data Base has been defined to store ETAM programs, project data, and data 
extracted from other Navy data bases to be used in specific evaluations. 

ETAM WORKING DATA BASE. There are three major sections of this data base. 

Program Library . The first section of the ETAM Working Data Base is a program 
library containing trie major programs and routines to be called during the 
processing operations. The programs contained within this section of the data 
base will be the: 

1. Description Structuring Program 

2. Range-of-Effect Program 

3. Assessment Proaram 

4. User Interactive Program 

5. Data Rase Maintenance Program. 

Programs will be stored as load modules in a partitional data set. A direct 
access storage device is recomiended for storing all programs, routines, and 
subroutines which are a part of the program library; as well as for storing 
project data, conversion tables, abbreviated records of entities for search 
retrieval, and extracted data from Navy data bases. 

Project File . The second major section of the ETAM Working Data Base will con- 
tain all project ddLa loaded initially using the Description Structuring 
Program, as well as any additional data developed as a result of performing the 
ETAM operations. These data will include: 

1. Project Descriptive Data (from Task 1) 

2. Alternatives to the Innovation (from Task 2) 

3. Risk Factors and Risk Reduction Projects (from Task 3) 

4. Decision Tree Factors (from Tasks 4 and 6) 
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5. ' Range-of-Effect Search Descriptors (fronrTask 5) 

6. Benefits Patterns (from Task 5) 

7. Cost-Benf1ts Results (from Task 6) 

8. Financial Analysis Data (from Task 7) 

In addition, abbreviated records of all courses, jobs, and equipments will be 
maintained as a part of this data base. These records will be appropriately 
subsetted (e.g., courses by phase, etc.) and Indexed to permit search once the 
search descriptors have been formulated from Task 5. 

Project data will utilize an hierarchical structure with variable length records. 
A dataubase management system* which provides interactive capability (e.g., 
RAMIS®*) is recommended to facilitate modifications to the analysis structure. 
Relationships between primary project data, alternatives to the primary project, 
risk reduction project data, and entities being cos-ted and aggregated will be 
maintained through external tables. The components of the project file; e.g., 
name, description, objectives, targe^ applications, empirical data, etc., are 
defined within the ETAM procedures in Section III of sfKIs report. The ETAM 
procedures should serve^jfs a guide in development of project descriptive data 
and risk reduction project data. 

Extracted Data . The third section of the ETAM Working Data Base will serve as 
a repository for the full set of data extracted from searches of master Navy 
data bases. These data will be maintained throughout the assessment operation 
and finally summarized and stored as a part of the project data. Individual 
hierarchical data sets will be established for entitles extracted from existing 
Navy data bases. Because many of the attributes of courses, jobs, and equip- 
ments are subject to frequent change, once extracted they will be used for 
processing in relation to a single proposed innovation being evaluated. The 
entity will be repeated in a separate data set if It is related to more than one 
project. New extraction may be required if the analys-is spans a substantial 
time period or if the entitles are related to a major new program which is being 
subjected to frequent change. 

Course Abbreviated Data Base Structure . The primary use of the abbreviated data 
bases is for searching out entities which match the descriptors .formulated 
through the assessor's analysis of the target application and interactive 
prompting using the taxonomies related to the entity class (e.g., courses). 
Figure iy-2. shows a recommended structure for courses to be maintained within 
the project file area of the ETAM Working Data Base. Additional predevelopment 
study has been recommended in Section I of this report to develop interim 
procedures for translating the ETAM nomenclature Into descriptions of course, 
tasks, and instructional vehi cle. attributes . This is Intended to establish 
the course indix design that will result in a reasonable economic tradeoff 
between storage requirements and retrieval validity. 



*RAMIS^ is a data base language developed by Mathematica, Inc. 



4 

f^Mi Ow<y A«f 



TAEG REPORT NO. 12-3, 

i 



LEVEL 1 



CRSNO 


— 

CRSNAWE 


— 1 


LEVEL 2 (REPEATING) 


1 


PHASE 


INDEX 



FIELD NAMES 

CRSNO: COURSE COP DESIGNATION 
CRSNAME: COURSE NAME 

CCMM: . COURSE CONTROL MODEL rvlANAGER 

phase: phase of course 

INDEX : descriptor TAG FOR EACH PHASE OF COURSE 

- STAGE OF LEARf\JBNG 

- TASK STRUCTURE ELEMENT 

- JOB CLASS 

- CONTENT TYPE AWD SUBSET 



FIGURE IV-2. EXAMPLE COURSE ABBREVIATED DATA BASE STRUCTURE 



LE VEL 1 

RATE/ 
RANK 



NEC/ 
NOBC 



DESCRIPTIVE 

DATA 




LEVEL 2 (REPEATING) 



TASK ID 



INDEX 



FIELD NAMES 



RATE/RANK: 
NEC/NOBC: 

DESCRIPTIVE 

data: 

TASK id: 
INDEX: 



ENLISTED RATING AND GRADE OR OFFICER RANK 

i>iAVAL ENLISTED CLASSIFICATION OR NAVAL 
OFFICER BILLET CODE 

RATE/RANK NAME OR S^EG/NOBC NAME 



JOB/TASK DESIGNATION 

DESCRIPTIVE TAG FOR EACH JOB/TASK 

^ TYPE OF SKILL 

TASK STRUCTURE ELEMENT 
- TASK SUPPORT CHARACTERISTICS 



FIGURE IV-3. EXAMPLE JOB ABBREVIATED DATA BASE STRUCTURE 
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Job Abbreviated Data Base Structu re. As with the course tbbrevlated dit« base. 

the job abbraviated data base wlTfl)© used to sea?^ch job/task entitles which 
match descriDtors fonnulated from the tixonoinles for the .lob/task entity class. 
The recommended additional predevelopment study In Section I of this report 
should provide the final specifications for data base design/ A preliminary 
recowRsnded structure is outlined in Figure IV-3. 

Equipment (Vehicle Type) Abbreviated Data Base Structure . The Inventory of 
equipments used within naval training would be maintained In the third 
abbreviated data b(i?e. Descriptive tags would contain 1) vehicle type, 
2) Instrjctional function, 3) utilization functions, 4) class of training 
bbjectlve, and 5) where used data. Specifications for the data base structure 
are intended to be defined in the recommended additional predevelopment stu4y 
identified in Section I of this report. 

DESCRIPTION STRUCTURING PROGRAM (DSP). The objective. Inputs, processing 
functions » and outputs for the DSP are outlined as follows. 

Objective . The DSP establishes a new project region within the project file 
section of the ETAM Working Data Base. It accepts the total range of descrip- 
tive data on the new project (proposal training innovation) and formeti It for 
subsequent processing. It also accepts major data subsets associated with the 
project which will Include: (1) experimental studies with descriptions ind 
resultant data; (2) risk reduction implementation projects with descriptions, 
resource requirements, schedules, and Impacts upon overall project success; 
(3) alternative project descriptions; and (4) unique costing data. All major 
data sulpsets will be associated to the overall project under analysis through 
special relational tables. The table structure will allow cross-referencing 
of data subsets for subsequent updating and restructuring of selected data. 
All data will be edited by the DSP and default data assignments will be made, 
, if possible, for any missing data. Input to the DSP will define the initial 
processing sequence .and output requirements. 

Input . Project file initialization and most updating will be performed via card 
Input. The major input data sets will be: 



1. 


Project descriptive data 


2. 


Alternative projects descriptive data 


3. 


Risk reduction project data 


4. 


Cost models Input factors 


5. 


D;5cis1on tree psrasmtars 




Job control instructions. 



Processing Functi ons. The DSP will accept input data sets and job control 
instructions. " ^rPTnput will be edited for missing md/or potentially errone- 
ous dQta. Job control instructions must Identify the manner In which missing 
and erroneous data input will be handled; the DSP ^111 perform the necessary 
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processing. The required file management routines will be called and assigned 
to process the input which may be new data or replacen^nt data. Once data sets 
have been assigned with edited and default data inserted, a project table will 
be created which identifies all data sets related to the primary project being 
proposed. \ 

Outputs . Two major output data sets result from processing the DSP. One out- 
lines errors and default data assignments.. The other presents the requested 
and generated job sequence instructions to the Executive Control System to 
control subsequent processing. 

RANGE-OF-EFFECT PROGRAM (REP), The objectives, inputs, processing functions, 
and outputs for the REP are outlined as follows: 

Objective . The two primary objectives of the- REP are, 1) to create. the necessary 
set of data base search parameters for obtaining the entities (e.g., courses, 
jobs) to which the innovation has potential application> and 2) to structure the 
benefit pattern used by .the Assessment Program in calculating the decision 
variables. 

Input . Project descriptive data from the project file will provide the t-arget 
applications for the proposed innovation. This will establish the initial 
descriptor content. Benefi t estimates and the empi rical data will define bene- , 
fit patterns and allocation rules for further modifying the search parameters. 

Processing Functions . The appropriate abbreviated data base will be searched 
based upon the target application and the descriptor index will be presented 
for the review of the assessor. A conversion table will translate numeric 
indices to appropriate .elements of the taxonomies related to the entity class 
for the target application. These same tables will permit the assessor to 
continually work with Fngiii,;, language descriptors' during the descriptive index 
refinement process.' RoutinS'i within the User Interactive Program (UIP) will 
prompt the assessor and select taxonomic menus to guide, this refinement process. 
Multiple indices may be developed if imprcvenients of the taxonomic menus are 
OR'd together. Once the descriptor indices have been forniulated, the appropri- 
ate abbreviated data base will be searched to obtain all entities which match. 

Allocation rules are for the purpose of furthen refining the list of selected 
entities. For example, if courses having a length of one week or more can 
benefit from the proposed innovation ^ this would become an additional selector 
factor when the list of entities is forwarded to the Navy data base controlling 
function for extraction of the full range of attributes associated with the 
entities. The assessor must determine if the additional selectors are to be 
used to limit the numuer of, entities extracted, or whether they should be 
applied to the total set of extracted entities oiice they have been loaded in 
the extracted dat^ section of the ETAM Working Data Base. This determination 
would be based upon whether the entities eliminated by the selector would ever 
be brought into the Assessment Program analysis from a cfiange in allocation 
rules, 

A final processing function of the REP is the formulation of the benefit pattern 
used to input various routines with the Assessment Program. The ETAM procedures 
for Tasks 4, 5, and 6 should provi'de sufficient guidance on the content and 
format. Some benefit patterns will involve directly cos table 'functions (i.e., 
straight displaceable costs) such as in the Ccise v/here equipment is ^eliminated 
replaced. 
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Output , The Range-of-Effect Program develops an output tape containing a list 
of entities and selector factors to be used in searching existing Navy data 
bases to obtain the full attribute set for each entity matched. The second 
output wi^n be a table of Variables, degree of change. Importance, confidence, 
etc.> which will represent the benefits pattern to be used within the Assess- 
ment Program. Allocation rules whicP were- not used in generating selector 
factors on the entity extraction tape will also be stored for use by the 
Assesment Program. 

ASSESSMENT PROGRAM (AP). The objectives, inputs, processing functions, and out 
puts for the AP are outlined as c follows. 

Objective . The AP provides the data^manipulation and computational capability 
for generating costs, benefit quantities, decision variables, and sensitivity, 
factors'. These become the key inputs to the decision maker. The AP will have 
the capability of aggregating and weighting entities either for inputting cost 
models or resulting from processing by a cost model. Decision variables calcu- 
lated on aggregated data will be obtained from either a cost-benefits analysis 
using a decisi.on tree framework, or from a financial return-on-investment 
analysis.^ 

Input . Job control instructions passed to the Executive Control System will 
guide the initial pyt)cessing sequence for the AP, thus the input factors 
selected will be a function of these instructions. Additional processing 
instructions will be provided by the assessor through input to the User Inter- 
active* Program. Each routine (training cost model, job ^cost model, decision 
tree generator, etc.) will require a set of data variables which will be stored 
in the extracted data section of the ETAM Working Data Base. Each rouTtine will 
also have provision for the generation .or selection of default data in the 
cases where information is missing. The individual requirements for the rou- 
tines with the AP are outlined in the "processing functions" section which 
follows. 

Processing Functions . The processing design should be guided by the ETAM 
Procedures, Task 6 and 7. The process in'g routines within the AP are: 



1. 


Training Cost Model 


2. 


Job Cost Model 


3. 


Equipment Cost Model 


4. 


Decision Tree Evaluator 


5. 


Economic Analysis Routine 


6. 


Sensitivity Analysis Routine 


7. 


Aggregation Routine. 



As previously mentioned, each routine will perform the data editing and default 
data selection for its respective data sets. 
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Training Cost Model . The Navy Training Analysis and Evaluation Group, Orlando, 
Florida, had developed a model for costing training requirements as a part of 
a TECEP Model (Training Effectiveness-Cost Effectiveness Prediction Technique) 
for evaluating training media. The basic logic of this model was incorporated 
into the Training Cost Model specified for ETAfvi. A program listing, flow- 
charts, and sample outputs from this program are included at the end of this 
section. Several features were incorporated into the model which are intended 
to suggest sophistication which may be introduced into a final developed 
program. 

1. Extensive input data editing and'default datci specification. 

2. Linkage between the cost model and the output of a previously developed 
Training Process Flow Model. 

3. User selectable policies for equipment purchases and equipment depreciation. 

4. Selectable output reports. 

5. Output format design compatible with CRT display to facilitate any con- 
version from the present batch to an interactive system with display 
capability. 

Job Cost Model. Th? cost model reconmended here provides a consistency with the 
proposed task descriptor discussed as a part of the job abbreviated data base. 
Figure IV-4 shows the Job/Task Cost Flow as it relates to the tasks which were 
matched with the search descriptors formulated in the Range-of-Effect Program. 
Those naval personnel who perform the tasks would have been extracted from a 
Navy personnel data base and stored in the extracted data section of the ETAM 
Working Data Base. Allocation rules (which have not alreaoiy b&en applied in 
limiting the entities extracted) may provide further selectivity in terms of "who 
performs task," or may be applied later in the costing sequence. For example, 
an innovation may apply only to tasks which are performed with a frequency of 
less than one per month. Thus, certain tasks will be eliminated aL this sub- 
sequent stage. Figure IV-4 shows the major cost determination paths. Each 
would require tijat job data beyond what is presantly available be maintained. 

Path 1 requires "time per task" and "task frequency" data which, multiplied 
together, provide a "change in performance time" which can be ascribed to the ' 
nnovation. Perfonnarrce time is converted into numbers of personnel which are 
now costable using personnel cost factors. A BiTlet Cost Model presently under 
development for the Bureau of Naval Personnel can be used or will have generated 
cost factors for making the detenni nations of total costs along this path A ■ 
description of the Billet Cost Model with a list of the variables used in 
calculating billet costs is reproduced at the end of this section of the .report. 

Path 2 provides for cost datermi nation when the innovation impacts errors which 
are made in the performance of a job or task. Errors can be defined as any 
deviation from criterion performance. At the present time, the costing of this 
path is visualized as a manual activity requiring a study of the specific error 
types affected by the innovation, and the determination of costs associated 
with the errors. 
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Path 3 IS for assessing costs associated with job support functions This 
would ijclude facilities, tools, test equipment, etc., all of which support the 
job performance. Estimating the quantities affected by the innovation is again 
v.isuanzed.as a manual activity, perhaps requiring a specific study. Costs for 
many of the job support items would be readily available through the appropri- 
ate naval activity responsible for their procurement and/or maintenance. 

The approach to job cost modeling outlined here precludes a need for extensive 
parametric studies which would be required if variables at the next lower 
hrncoH^ ^'^r" ^'"Ji^oduced. There are, however, existing models that will 
An Air Fl^rp'°^% °^ personnel pf of perfonnance activities. 

An Air Force study* looked at several alternative methodologies and dost models 
[SLfrr^h'^'-'°%\°^ on-the,job training. Thus, if an'innova'tion was 
targeted at changing this job' component (as in the Scenario Number One in 
cost'effects •^^P°'^*)' ^ similar approach might be taken in determining the 

Equipment Cost Model. The "instructional vehicle" route in the Range-of-Effect 
lask b, is taken when the innovation is concerned with a straight cost displace- 
ment effect. The data base accessed for obtaining.attributes of equipments 
should have cost factors available. The model for determining equipment cost 
would be similar to the equipment handling portion of the Training Cost Model 
Relevant variables would be: . . nuuci. 

Equipment purchase cost 
Purchase policy 
Equipment maintenance cost 
Life of equipment 

Depreciation policy. ^ 

Decision Tree Evaluator. The decision tree evaluat-ion ro"t-ine performs the 
necessary calculations required to fold back the tree and determine the value 
-of the decision variable. A program for performing such an evaluation is 
presently in use within DOD. The Decision -iTree Language System* was used to 
program the standard decision tree developed for ETAM. A copy of the oroaram 
as well as copies of several program runs using various features of the oroqram 
are contained at the end of this section, of the 'report. Program, 

The inputs to the decision tree evaluation will be able to be entered directly 
in the batch mode or interactively to the User Interaction Program. The inputs 
V to this> routine are: . iM^iuLi 

\ ■ ' 

Outcomes A through D 
Success Probabilities I, U through Z 
Risk Package Cost. 

\ 

\ 

\ 

^b^mers, B N, , Dunham, A. D. and Nordhauser, F. 1974. The Develooment of a 
\ Methodology fo r Estimating the Cost orAir Force On -thP- .lnh T^^lLf ' 
\ Report No. AFHRL-T R-y4-:34 , Contract F41609-7^-C-0048, fV Air For l^" ■ 
Human Resources Laboratory, Brooks AFB, Texas. ' h H.r r.urt.e 

**A program developed by SRI (Stanford Research Institute), Menlo Park, Ca 

available on the DARPA Network (ARPA/HRRO, Arlington, Va) - 
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The program written as a part of this study does not contain the facility for 
bS^eSll"* iljSde""^ ''^^^ reduction package, however, this addition can 

••v • 

Economic -Analysis Routine. Since the ETAM procedures' in Section III. -Task 7. of 
Inll^fZil r'l^' sufficient guidance for this routine, and sinJe the?l are 
number of conipuT:er programs available for-making these types of 
calculations, no additional design specifications are included here. 

Se'ns-itivity Analysis Routine. Two sensitivity analysis routines are required. 
i^. lLTfu!'il .^^^ decision tree where the decision variable, is evaluated for 

Break-even value 
Decision change (from/to) 
Sensitivity factor 
Probability factor. 

lincl^f^r^"^ and confidence range for each variable, are the inputs to the 
sensitivity analysis routine. - r « 

A second sensitivity analysis routine (which can actually be developed as a • 

■ vi^^?^W^.°^^^^^''^'^ V^^ 'T?' rate-of-retum and pi^sent value dec s?on 
fn ll l dependent variables, and will allow a sensitivity calculation 

to be perfomed using each benefit variable and discount-rate as Independent 
A report ^similar to that proposed for the sensitivity Analysis on 

;e?e??e^^io°^■o^"L^;gS^^'uidan^c""'^'• ™ P""'"^^^' Subtask'07.0/should be 

Aggregation Routine. The assessment program must have the facilHty for summing 
?!^nJ?c"'H.°^ evaluating multiple entities. For example, if the innovation 
impacts Tifty courses, each course would be processed through the training cost 
model, and the results aggregated through this routine. Similar aggJegations 
would be performed for jobs^ equipments, etc. , aggregations 

M£^.;The AP routines generate output data sets which can be accessed by the 
^tMn '^hl^',*^°"^ fP^I- generation, and output structuring routines 

"^oll^^l^^^ ^"P-- Pressing function.. 

9^J^ff^^g - The purpose of the UIP is to maintain control over all processing 
functions and input and output both batch and interactive. It serm as the 
Executive Control for the programs, routines, and data bases. t also controls ' 

n ?%o;^ tt' ng"1n"?h-'' '''''^l'' P^'-Pting. dajfretHev" 

!ith?n ?h! " ' S""""^'^ description of the routines 

w thin the UIP is provided, however, final design specifications require addi- 
tional study as recommended in Section I of this repopt. 
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Inputs . Inputs to the UIP can originate from several sources • Job control in- 
structions from the DSP generate the calling sequence for processing the program 
routines. These can be overridden by terminal input. Both the job control 
instructions and terminal inputted commands are analyzed and processed by-the^ 
Executive Control System within the UIP. Input data*set§ for major routines a>e 
accessible through the UIP in order that data changes^can be .made when formulatrV 
ing "what if" type questions. The UIP also accesses these data sets,, as well as/ 
any output data sets from the routines, to perform data retrieval^ and output 
report structuring. ^ 

Processing Funttions . The primary processing function of the UIP is to handle 
commum cations and control for data 'handling and processing. An Executive Con- . 
trol System containing a skeleton monitor and communi cations. moni tor is the 
focal' point for this function. The following functions are also performed by 
the UIP. 

1. Data input and input editing. This function is performed for input 

vi a the terminal . ^ ■ - " • ^ ' 

2. Prompting. A number of prompting routines including assi stance "help") 

to the user, will guide the assessor in data input and retrieval , processing 
control, and output structuring. ' ; * 

3. Output reporting. The UIP processes the data sets outputted .from the 

. various routines, structuring the data into standard report formats as 
outlined in the ETAM procedures in Section III of this repqrt. 

Output . The major outputs are data exception reports from the data, editing 
function, prompting instructions to the assessor when operating interactively, 
an interaction log which provides a record of terminal requests, graphic type 
reports (if feasible, based upon the results of 'the istudy recommended in 
Section .1 of this report),!and standard output reports. 

DATA BASE MANAGEMENT PROGRAM (DBMP). Standard routine^ within this'program 
perform initial data base loading, data base editing and updating, and data 
dumping. As a part of the data base maintenance function, error arid activity, 
and maintenance reports will be provided to the user. These programs operate 
upon the elected data base management system. 
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PROGRAM - TRAINING COST NOOEL 

TECE^ COST HOOEL AO.^^TATlONS FOR ETAH lY L. OUFFY 
DATE - 4/21/79 



PROGRAM SCTU^ 

REAL INTSW, IMOEV,l/sCS^O,N$C$TO, INONNT, INSTSL , IN^LAT* INMATE 
REAL NSTUO.NTCAVtNlNSTR,NAOM|N.NFAClL.NSU^^LfNMISC,NE«UI^.NIHO 
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INTEGER GRAOt^URCHF.AEQUIf.OEfF 

INTEGER 0IGITI20I / 1 • 2 f If 4 , 9 , ft »7 « I • «» iO» 1 t » 12 » 1 1» 14 , 1 9» 1ft » 
I 17.1l,l«t20/ 
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DIMENSION A0il(20) 
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2.FU1H0.0#0ATI,IU^$F0.1UFIT0.«M:»F0,«C$T0.PI1P » ipw.* 

NAHILIST/COSTF/lTUOSL.STCSTl.lTCSTl.miTlL.AWdL.IOOWTCMTKH.CFl 
10fT,ClH0.|y|H,C»UFF.CHIl. ORATE. INKATI .eu«.<n. i rvn, t^i 

MANILIIT/OIKOF/CRAO.COFNT ' 
REAOiStTICHFI 
RiAOOtCOSTFt 
RiAOI9,OINOf > 

rJJ2i?*"°°' •^T1.RFT2,RFTI.RFT^.RFT1.RFT*,MT7.LAST 
4000 FORMAT IlLl) 
^ IF IN. NI. 99999) CO JO }0 

C RBAO T^F KOOIL OATA 

29 RCAO (10»*020«eNO-30) C OUR S XiNX « ARATIX » TLINIX «RCRATX . AaCVTV .MURkO ( 
nKI«MDl2^WRA0Cl»,A0*xM>.A0IX(2I.AM!im 
ft020 FORMAT ( A*. 1 2 . F^. »,F^ . 1 ,F4. J.F4. 1 , II », If i . 1) 

IF ICQURSX.NI. COURSE) CO TO 29 

N - NX . 

ARATB - ARATEX 

TLENtH * TLJNBX 

RCRATE - RCRATX 

ARCYTH - ABCyTX 

00 28 I-1,N 
28 BAAOl 1 ) - NORAOl I ) 

C 

C RBAO NISSINC OATA ANALYSIS AHO OIF&ULT ROUTINES 

10 IF IN.OT.O.ANO.N.Li.20| 60 TO 40 
M - 20 

CALL NiSSfiC tSOil) 
40 IP U«*TE.6E.O. .ANO.AIIATE.lt. 1.) tO TO 90 
ARATE - 0.0 
CALL N1SSN6 I40i 1 ) 
to IF lTLEN«H.BT.0..AN0.TLlNeH.LT.200. ) CO TO 40 

CALL NISSN« ItOfO) 
«0 IF CRCRATE.il. I). .ANO. RCRATE. LT.l.) 00 TO 7fi 
RCRATE* .l«ARAVE 
CALL ttlSSNt UOil) 
70 IF IABCYTN.iE.0..A«0.ARCyT«.LT.200.) 60 TO 90 
IF (RCRATE.EQ.O. ) 00 TO 80 
IF ITLEMH.LT.2.) %0 TO 80 
CALL NlSlNO (TOtO) 
60 TO 90 
80 ARCYT)^ I.Q 

CALL NlSSNt (TOtl I 
90 IF (»SCHOF.iT.0..AN0.)*lCHO#.LE.92.) 60 TO 100 
WSCMOF- 98.0 
CALL NliiN* («0*1) 
100 IF (TSFatO.II.O..ANO.TSFOSO.LT.l.) tO TO 110 
TI#0IO« 0.0 
CALL NISSN6 I lOO* 1 I 
110 IF (FI1F.6I.0..AN0.FBSF.LI.1.) 60 TO 120 
FilF - O.l 
CALL NISSNS (llOi t) 
120 IF nNTSF0.6E.0...AN0.INTSF0.Li.l.) «J TO UO 
IMTIFO* 0.0 
CALL MISINO niOil) 
110 IF |ANTSPO.il.O..ANO.ANTaFO.LC.l.) 60 TO UO 
AMTIFO- 0.0 
CALL MtSSNQ MSOtl) 
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1 ^0 


ir 1 C U 1 J P U • uC • U • • aNU • C U 1 ) r U • L C a I • 1 \t\J lO 140 


22i 




IF ( E QUNT C • NE • 99999 ANO aE QUN T C ■ NE • 0 • ) CO TO 190 


224 




IF 1 LOF £Q .Nfa 99999 ■ ANOI LOFEQ* NE , 01 GO TO 190 


225 






224 




CALL HISSNG 1140. 1) 


22T 




r.n TO l&o 

UU 1 U 1 




1 «n 




22f 


in 
140 


IF (PUMCHF. CE • 1 . ANOaf URCHF • L T • 1 0 ) GO TO 170 








211 




CALL HISSNG Il40«ll \ 




70 


IF (OEPFbCE . 1 . ANO.OEPF. LT. 10) GO TO 1 tO 


233 






2 34 






21S ■ 


1 10 


IF (LOF E Q. GE • 1 . AMO. LOFEQ.LE • 90), GO TO 200 






i£ tCOTCBA MC A 1 rA ta ■ aA 

Ir 1 c U I iPUaflc • U ■ I TO 190 


21T 




(EQUNT C«NE • 99999 • .ANOa EQUNTC aNE aOa ) 60 TO 190 


231 




LOFEQ * N 


219 




CALL m ssNG lltOill 


240 




tin TO 9nn 


241 


190 






200 


IF ( SQF TST.GE. 0. . ANO.SQFTST .L T • 1000. ) CO TO 210 






SOFTST" 0,0 






CALL HISSNG (200*1) 

IF (SQFT iN.CE. 0.. A)^0. SOFT IN. L Tl 1000. 1 GO TO 220 


245 


2 10 


244 




SQFTlN" 0.0 


24T 




Call hissng I2i0ti) 




220 


IF ( SQF T AH.GE. 0. . ANO.SQFT AH.L T . 1 000. ) GO TO 230 


249 




SQFTAH" 0.0 






Call hissng i220«l) 


i 


230 


IF I i HOOE V.GE .0. . ANO. IHOOE V.L T. 1000. ) CO TO 290 


2 J2 




IF I PUI HO.GT. 0. . ANO. PUI HD.L E . t. 1 GO TO 240 






IF (UPOaT E.GT .0. . ANO.UPOATE .LE. 1 . ) CO TO 240 






IHOOEV" 0.0 


7«« 




Call hissng i230*l) 


?«A 




rA T A 7 ■ A 1 


?«T 




CALL HISSNG 1230*0) 




250 


IF 1 PU I NO. GT . 0. . ANO.PUl HO.LE. 1 . ) GO TO 270 






IF 1 1 HOOE V.NE .0. ) GO TO 240 


240 




If IUFOAT E .GT .0. ■ ANO.UPOATE .L E . 1 . ) GO TO 240 


24 1 




■ 111 MA M 1 A 


242 




CALL, HISSNG I290»l) 


24) 




tin T A 7 T A 




^ trt 


Call hissnc i290*o) 


24S 


270 


IF (UPOATE .GE. 0. , ANO.UPOATE.LE . 1 . ) GO TO 290 


244 




IF 1 IHOOE V.NE .0 . 1 GO TO 210 


247 




IF (PUIHO.GT .0 , . anO.PUIHO.L T. 1. ) CO TO 200 


241 




UPOATE" 0.0 


stn 




Call hissng 1270*1) 






CO TO 290 




2t0 


CALL HISSNG 1270*0) 


917 


290 


IF 1 SUPSPO. GE .0. • ANO.SUPSPO.l T • 1 000. ) GO TO 300 


2T1 




SUPSPO" 0.0 


274 




CALL HI SSNG 1 290* 1 ) 


27S 


300 


IF (SUPSTO.GE.O.. ANO.SUPSTO.lt. 1000.) CO TO 310 


274 




SUPSTO- 0.0 


277 




CALL HISSNG 1300*1) 


271 


3 10 


IF IHSCSPO.CE.O. .ANO.HSCSPO.lt. 1000. > GO TO 120 


279 




HSCSPO- 0.0 


2t0 




CALL HISSNC (310.1 ) 


211 


320 


IF IHSCSTO. GE.O.. AHO.HSCSTO.lt. 1000. ) GO TO 110 


2t2 




HSCSTO- 0.0 


213 




CALL HISSNG 11^0*1) 


214 


330 


IF (STUOSL. GE.O. .ANO. STUOSL.lt. 90000.) GO TO 340 


2IS 




9TUGSL*' 0.0 


214 




CALL HISSNG M30. 1 1 


217 


340 


IF ISTCSTl. GE.O. .AN0.STCST1.lt. 10000.1 GO TO 390 


211 




STCSn- 0.0 


219 




CALL HISSNG (340*1 ) 


290 


390 


(F (SrcST2. GE.O.:. ANO. STCST2.lt. 10000.) GO TO 340 


291 




STCST2" 0,0 


292 




CALL HISSNG 1 390* 1 ) 


293 


340 


IF 1 iNSTSL. GE.O. .ANO. INSTSL.LT. 90000. ) GO TO 370 


294 




(NSTSL- 0.0 


299 




CALL H(SSNG 1340*1) 


294 


370 


IF (ADHSL. GE.O. .ANO.AOHSL.lt. 90000.) GO TO 340 


297 




AOHSL - 0.0 


291 




CALL HI SSNC ( )T0* 1 ) 


299 


340 


IF (EQUNTC.CE.O. .ANO. EQUNTC.NE. 99999. ) CO TO 400 


300 




IF lEQTSPO.NE.O. ) GQ TO 390 


301 




EQUNTC- 0.0 


302 




CALL HISSNG ( 310* 1 1 


303 




GO TO 400 


304 


390 


CALL HISSNG OtO.O) 


30S 


400 


ins IPCTPCH. GE.O. .ANO. PCTPCH.LE. 1. 1 GO TO 4l0 


304 




PCTPCH- 0.0 


307 




CALL HISSNG (400*1 I 


301 


410 


IF (CPSQFT. GE.O. .ANO. CPSQFT.NE. 99999. ) CO TO 420 


109 




CPSCFT- 0.0 


310 




CALL HISSNG (4lO*I) 


3 1 1 


420 


IF <C(nO. GE.O.. ANO. CIHO.LT. 90000.) GO TO 430 


312 




CIHO - 0.0. 


313 




CALL HISSNG (420*1) 


314 


430 


IF lEVtH.GE.O.. ANO.EVIH.lt. 1. ) GO TO 440 


314 




EVIH - 0.0 


314 




CALL HISSNG (4l0f 1 ) 


317 


440 


IF (CSUPP.GE.O.. AND. CSUPP.NE. 99999.) CO TO 490 


311 




CSUPP - 0.0 


119 




CALL HtSSNG (440. 1 ) 


320 


490 


IF ICHIS.GE.O. . ANO. CHIS.NE. 99999. ) CO TO 440 


32i 




CHIS - 0.0 


322 




CALL HISSNG l490*l) 


323 


440 


IF iOKATE. GE.O. .ANO.OKATE.lt. 1. 1 GO TO 470 


324 




ORATE - 0.10 


329 




CALL HISSNG 1440.11 


324 


MO 


IF (INRATE.CE.O..ANO.INHATE.LT. ». ) GO TO 400 


327 




INNATE- O.Ot 


321 




CALL HISSNG (470.11 


329 


410 


IF IGKAOIN). GE.O.. ANO. GRAOINI.NE. 99999. 1 CO TO 490 


330 




CALL HISSNG l4IO,0) 


33) 


490 


IF ICOPHT IN). GE.O. .ANO. COPHTlN).NE. 99999.) GO TO 910 


332 




DO 900 |"l.N 


333 


900 


COPHT in- 0,0 
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• 

ISf 

IIT 
lit 
lit 

Ml 
Mt 
Ml 



111 
992 
191 
1D% 
19» 
114 
IIT 
IM 
119 
MO 
Ml 
Ml 
Ml 



14 T 
Ml 

MO 
ITD 

in 

1T2 
IT). 

375 
IT6 
ITT 
1T« 
IT* 
MO 
)QI 
lit 
Ml 
114 
119 
104 
117 
IM 
190 
190 
191 
19t 
191 
194 
199 
194 
107 
199 
199 
400 
401 
401 
401 
404 
409 
404 
407 
400 
400 
410 
411 
4lt 
4.11 
4"14 
419 
414 
417 
418 
419 
4t0 
411 
411 
4111 
4S4 
419 
414 
41 T 
419 

410 

41* 
4fil 

411 

411 

414 

419 

414 

417 

410 

4S9 

440 

441 

441 

441 

444 

44* 



CALL nlStM (440, I ) 
910 1^ l»}Q30.E0.n to TO 900 
IF (N. 91.201 60 TO 919 
lH0fiX9 - H*\ 
00 919 I«INO9il9*10 
COMTIl 1-0.0 

919 MAOIII - 0 I 
919 COdTlKUC 

KOftQL IQUATIONI - TCCMftlCAL f4GT0llS 

UIM9TM - aef««miMoeffv*fvipio«40.o 

CUIMO • UlnOTM^lMO 

KVIM m CUIP«*^VIM 

•0 giO I-liN 

STUOin - 9«A0(n/(l.0<4tATtt 

I9«AD - iMAO^eilAOin 

AASIN * AAStN«STUO( 1 ) 

ITUOMWC n- irLiN«mSTUOI I M«U.0-.9«A«AT|) 

IMttWICdl- llCI(ATf«STUO(n«AIICVTM 

40911) - <STU!W«n)*MMIIIC(n)/M|CMOF 

0409 - 0409 ♦ (AOttlHAOtXdU 

S409 - SAOt ♦ 409(1) 

AA«9 - 4409*409(1 ) 

PSPd) - MA09in/( l.0~Tlf010l l«(|.0VHtP) ) 

lllNSTlKn- PSPin«lNTlPO 
TINITR - TINITII^IIINITIKI) 
IIADNIN( I )- PSP(M«4I(IIP0 
TAOMIir - TADfflM«IIAOMlN(l) 
MQUl^d}- (»SPi I )«iOTSPO 
TIQUIP " TI(H<t^»Rl9UrF(n 

«P4ClLd »- UPd )«IS4f TST*INTIN}*t«f TdH4MTt»0«9«fT4l|) 

TPACIL - TPilClL*«PACIL(d 
IMONSITin- UlW}TH«UPOATe 
jmmj - TMD«*tlT* IMOIHIT{ I I 
MUPPLd )- PSPl d*IUPSP0*tTUOI d«SUf STO 

TtuppL^muppLin 

PSrd)«M|CIPOf|TU0(n«MICiT0 
JAOl^AOONd i 
TltlSC*llMlSC(r> 

) 0A09-04O9«i-l.t 



T$WL 
KM 1 SC d ) 
JAOO 
910 TMISC 

IF (0409. LT 



912 
9)4 



942 
944 



994 

990 



1409 - ((OA09/I409U.1»«100 

tF dA09.CT.5.AN0.JA09.Nl.0t CALL MliSN* (919.1409) 
AASIN - AASIN/N 
kdhkO - ICRAO/H 

A401 - AAOS/N « 

AP$P • IWSW/H 

AItt5T4 - TIMim/M 
'AAO«t|N - TAOHIM/N 
AlfiUP - TIQUIP/N 
APACIL - TFACIL/N 
AIHOUT - TMDIMT/H 
ASUPPL - TSUrPL/N 
AMISC - TMtSC/N 

THIS ROUTINC CALCULATCS IQUIPM9NT PUtCMSI A«tO OffMlC I AT 1 ON 
SCHIOULCS PROM USill SlLlCTIO POLICY OI*TtONi 

- PtMCHASi POLlCtll AN4tTlll 

GO TO (9)0«l40k990*94O«IT0f990«99C*400(4lbtfPU«CHf 

pOttCV t - PURCNA)! TOTAL AVflRAtf II9MTS 
IN FIRST VIAR OF lOOIP LlPI 

COHTiMUe 

DO 9)4 I«1,N«L0PQ0 

rSMPi * 0.0 

IMOEXl- LOPIO»I-l 
DO 9)2 J-I« INOexl 

TiNfi ■ REQuiPi j)*renpi 

COtlTtmjE 

PEQU|P(n.- TCKPI/lOPCQ 

COMTINUe 
GO TO 420 

POtlCY 2 - PUftCHAM TOTAt MftClNT OP MftlMUM 

CO«T.NU. »•« W U« 

00 944 I^lfNtLOE^eO 

TIMPl - 0.0 ^ 

INOCXI" LOrCfKl-I 

DO J- I , INOCXI 

IF dfMPi.LT.^{QiJip(jn rcMPi-af9uiP(j) 

CONTINUE 

PEQUIPd »- TINP1»PCTPCM 

CONTIRUE 

CO TO 420 

POLICY 3 - PURCMAil UP TO AVtftAOi RtMTl 9UT 

NOT ERCilOtieS P9MTS IM Vft OP HUkCH 

COMTimiE 

DO fsa l-l*NfLOPEQ 
Tt{S?l - 0.0 
i»OQNl- LOPlO^I-1 
00 992 >l* iNOIdl 

TEMPI » RRCUIPUUTiMfl . 

CONTimii \ ; 

PiOUlPin- TSMPl/LOPIQ \ 

IF (PEQUIPd). Ll.t^iOUIPIlM 90 TO 990 

TEFIPl - PiOUlPd ) 

INOEMI* LOj'tOtl'^l 

DO 994 J-I«I»0EX1 

IP (TEMPI. LB. 0. ) CO TO fu 

IF (TEMPI. ST. RCOUIP(JM PEQUlf ( J )-R|QU I P( J I 
IF ITENP1.LQ.R(!0UIP(J)) PEOUIP ( J )mT1M«» 1 
TEMPI - TEMPl-PiOUlP(J) 



CONTIMUf 
CONTINUE 
90 TO 420 



COt^TimJE 

DO i|4e l-lfNfLOPfQ 



- PU«CHAlE UP TO PCT Of MAR RMTi OUT 
NOT ERCElOtNC ftOMTS IN VR Of PORCH 
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TEHPl - 0.0 


\M 




INOEXt- LOFEQ»l-l 






DO 5*2 J-I 


» INOEXI 






IF <TE«PJ. 


LT.I)ftQU|P< JH TEMPl»IIEQUIf 1 Jl 


490 




5*2 CONTINUE 


491 




PEQUIPI 1 )- 


TEMPl*PCTPCH 


492 




IF (PEQUlPtn.LE.NEQUlPM)) CO TO 5*1 


493 




TENPl - PEQUIPM \ 


494 




INOEXl- LOFEQ»J-l 


499 




00 5** J«I 


fINOEXl 


49t 




IF ITENPULE.O, 1 GO TO 5** 


497 




IF (TEMPI. 


GT.NEQUIP< JM PEQU|P(J)»IIEQUIP(JI 


491 




IF iTEMPi.LE.XEQUIPUM PEQU 1 P ( J > -T CmP 1 


499 




TENPl - TEMPl-PEQUIPJ J) 


4fr0 




5** CONTINUE 




4ftl 




5*1 CONTINUE 




4»2 




CO TO *20 




4»3 




970 CO TO 930 




4*4 




510 CO TO 5 30 




44) 





590 CO TO 53D 




4*4 




*00 CO TO 5 30 




4*7 




*10 CO TO ))0 




4*1 


c 






4*9 


c 




- OEPNECIATION POLICIES ANALYSIS 


470 


c 






471 




*2D CO TO (*30 


.640,*50,**0, *70» *IO»*90t 700, 710> , OEPF 


472 


c 






473 


c 




POLICY I - OEPRECIATE LlHEAHLY OVEH 


474 






LIFE OF EQUIPMENT 


479 




*30 CONTINUE 




47* 




TEHP2 


LOFEQ 


477 




00 *32 I>1 


»LOFEQ 






TEMPI 


1 


479 




poepM) - 


TEHP1/TEMP2 


410 




632 CONTlNUe 




4t 1 




CO JO 7 10 








POLICY 2 - OEPiECIATE lY SUN-OF-OIGI TS 


41 3 






OVEN LIFE OF EQUIMEnT 


414 




*40 CONTINUE 




4t9 




SUM " 


0.0 


4t* 




DO *42 I-I 


, LOFEQ 


417 




SUM 


SUM* I 


4II 




*42 CONT INUE 




419 




POEPin - 


aOFEQ)/SUM 


490 




00 644 |-2(L0FEQ 


491 




pdepU) - 


aOFEQ-n/SUM»POEPt I-I I 


492 




*44 CONTINUE 




49) 




GO TO 730 




494 






POLICY 3 - SET DEPRECIATION SCHEOULE 


499 






TO FULLY DEPRECIATEO ALL YEARS 






*50 CONTINUE 




497 




00 *52 1-1 


»LOFEQ 


49t 




PDEpMI - 


1,0 • 


499 




*52 CONTINUE 




90Q 




CO TO 730 




90l 




**0 CO TO *30 








*70 CO TO *30 




903 




*I0 CO TO 6)0 




904 




*90 CO TG *)D 








7 00 GO TO *30 




90* 




710 CO TO *30 










90S 






- COST FACTORS • 


J?n 






> 






7 30 L 


N»l 


?! 7 




IF <L.CT.20t CO TO 750 


■ ! ^ 




00 740 I>L 


.20 


513 




CNAOin <• 


0 


9l4 




740 COPMTU) - 


0.0 


919 




750 CONTINUE 




91* 




TOTAEQ 


O.D 


91 7 




00 790 I-l 


(N 


9l 1 




CSTUOM I - 


I < STUOMMI n»SMMRRC M U / 5 2 ) • S TUO S L 


919 




CTNAVM) - 


STUOU l*«STCSTUSTCST2«U.O-.5«ARATEn 


920 




C INSTN M )- 


RlNSTR(n*lNSTSL 


92 I 




CAOniN< n- 


RAOHiNt 1 l*A0M5L 


922 




TPEQUP 


TPEQUP»PEQUIPI I 1 


923 




IN0EX4- PEQU1P( 1 U0.5 


524 




PEQUIPM )- 


INDEX4 


529 




CEQUIPI I >- 


PEOUl P( 1 )*EQUNTC 


52* 




LIFENV 


ULOFEQ-N-I 


527 ■ 




IF ILIFERV.LE.OI CO TO 7*0 


52* 




NVEQ 


NVEQ»CEQUIPM 1 *PDE P < L IFERV ) 


529 




7*0 CONTINUE 




530 




TOTAEQ 


TOTAEQ»PEQUIP( I ) 


531 




AEQUIPI 1 )• 


TOTAEQ 


932 




TAEQUP 


TAEQUP^AEQUIPd I 


53) 




LOISLT 


l-LOFEQ 






IF (LOISLT. LE. 01 CO TO 770 


535 




TOTAEQ 


TOTAEQ-PEQUIPtLOISLT 1 


»»t 




AEQUIP 11 ) - 


TOTAEQ 


« il 




770 CCQUlPin- 


CEQUIPU I»AEQUI Pll )*COPMTI 1 ) 






CFACI Lt ( )- 


RFACILI I )*CPSQFT 


5)9 




CIMOMT II)- 


IMOMNT ( 1 )*CIHO 


540 




IF (l.CT.l 


CO TO 775 


541 




ClWXIf { 1 )- 


CIMOMTI 1 UCUIMO 


94 2 




775 CSUVLIH- 


RSUPPLi n*csupp 


543 




cnisctl) - 


RHI SC 1 1 ) *CMI S 


544 


c 






545 


c 




- COST SUMMARIZATION 


54* 


c 






54 7 




OlSCNT,! n- 


(2.0»0RATE )/ r2.0«l 1 . 0»0R A T E 1 • * I ) 


54* 




INFLAT U )• 


r2.0*<1.0*|NRATE)**n/<2.0MNRATE) 


549 




NSTUD 


NSTUO*CSTUO( I I 


550 




0 S T UO 


OSTUO»CSTUOfn*OlSCNT(l) 


551 




ISTUD 


ISrUO»CSTUDt 1 J*INFLAT(I ) 


552 




FSTUO 


FSTuD»CSTUOn )*|NFLAT J H*DISCNT ( 11 


55) 




NTRAV 


NTRAVtCTRAVi I ) 


554 




DTRAV 


DTRAV»CTRAV< I)*OI SCNTI I ) 


555 




I I R AV 


ITRAV»CTRAVI li^lNFLAT < 1 ) 


95* 




FTRAV 


FTRAV»CTRAVn)»lNFLAT m«OlSCNTn ) 


557 




NlNSTR • 


NlNSTR«ClNSTR(n 
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t94 
99f 
f*0 
9*1 
M 
f*» 
M4 

9*6 
9*7 
9*1 

9*f 

.978 
971 
971 
971 
f74 
979 
97* 

. 977 
971 
9 79 
9«0 
9*1 
9tl 
9*3 
9t4 
9*9 
91* 
907 
9*t 
9*f 
9fO 
9f 1 
991 
9f» 
944 
9*9 
9t* 
9«7 
9«l 
9«« 
400 
401 
*02 
*01 
*04 
*09 
*0* 
407 
40t 

*0« 
410 
411 

612 

41S 

414 

419 

414 

417 ' 

611 

419 

420 

421 

422 

421 

424 

«2f 

42* 

427 

420 

42f 

410 

4>t 

412 

411 

414 

419 

41* 

417 

410 

*1« 

*40 

*41 

*42 

*41 

*44 

*49 

44* 

447 

441 

44t 

490 

491 

492 

691 

494 

499 

494 

497 

490 

499 

440 

441 

442 

441 

444 
449 
444 
447 
4«0 
449 



OIHSTH - 0INSTH«CmSTH(n»OlSCNT(il 

IlNJTa - MNlT«*CIMT»»tn»lNfLAT{n 

HH%JH - FtNSTH»C]N$TAU>*iNrLATIIi*011CN7in 

NAOHlN " NAOMIN«CAOniHl| ) 

OAOMIN - DA0NlN*CAm<IM(I)«OltCNTin 

lAOMlN • IA0I<IN4-CAMINII)«INFLAT(I) 

FAONIN ■ FA0MlN«CA0MIN(n*IN#LAT(n*0UCN7(n 

NfACiL • NPAClL^CFAClLd > 

OFACIL - OfAClL*CfACIHn«OIItlimj 

IFACIL - 1FACIL*CFACILI n«lP»^L4Hn 

fFACiL - FFACa»CJ»AClHn«I»>LAHn<*l»CHT<n 

f OSUPPL^CSUP^Ll n«0!tCN7<n 
0 IIU^^L - ISUPPL^CIUPPUU »«lN#t*Tt 11 

FSUP^L - FlUPPL»CJl>fPLlll»INFtAT(I»»OllCNTMt 

OMiiC - 0MMC*CMlSCtn«01SCN7in 
IMISC « |niSC»CHISCin«INfLA7Il I 

FMisc - FHisc*cniscj n«i»#tAT( n»oiicNT( n 

IF { I.NE.N) CO TO 710 
CEQUlPr n-CEQUIPf I loffVEQ 
CIMOMTI I )-Clt10nT 1 1 }-RVlM 
TtO NEOUIP - NeQUM«CEQUl(>< n 

OigulP - OEQUIP»CEOUlPin*OISCNTII) 
lEQUlF - lEgUIP^CEOUlPI n«lNFLAT 1 1 ) 
fEQUlP - F6QUI(»*CtQUlPCn«If«FLATU|«0ISCMT( 1) 
NiMO - MIMO^CIMOnTl I) 

OIMO - 0lM0»ClK0n7in*OISCNT(I) o 

IlMO - IlMOi^CIMOnTI I I*IM#LAT 1 1 ) 

FIND - FlHO*CmD«Tn)«OI5CNTn»«II»f«LA7in 

COPTOU I)- CSTUOU )*CTRAVJ| J*CINSTR(I)*CAOI<lNJn*CPAClUf n 
♦ ClMOnrU )«CSUFPLIII*CHIIC{I I^CfQUlPtl ) 
COFTOT I I )/CRAOC i ) 
COFTOTID/STUOd) 
COFTOTI I )/F$FI ! ) 
TFEQUF/N 
TACgiiP/H 

fi?AClU»FACO$T o 
QFACIL^FACOST 
IPACIU^FACOST 
FFACIL^FACOST 

tNA0C«NSTU0*ttTM/;V*NIHSTR*MAO*11H«»FAClL*NSU^PL*«MISC4Nt8UIP«NIflO 
S0AOC-0STU0*0TRAV»0IHSTRt0A0MlH»0FAClL«0IUf>PL*0MIIC«0f«UIM0f HO 
$IAOC"lSTUO»lTRAV*MNSTR»IAD«IN*IFAClU*lIUfPU«INItC*IIOUlF*M»0 
. SFAOC*FSTU0*FTRAV*FlN$Tn*FAD«IN*FFACIU*FIUPPU*FMI$C*F|(KJ|P*F|M0 
CNAOC-SHAOC/ICRAO 
COAOC-SOAOC/ICRAO 
CIAOC-SUOC/ICRAO * 
CFAOC-SFAOC/tCRAO 
UNAOC-SNAOC/ t AASlN*N) 
IU3AOC*SOAOC/I AASrN*N) 
UIA0C«SIA0C/(AAS1N*NI 
UFAOC«SFA0C/( AAllN*Nt 
FNA0C»SNA0C/1PSP 
POAOC«IOAOC/IPSP 
PlAOC-SlAOC/IFSf 
PFAOC-SFAOC/IPtP 
ORATf « DOATftlOO. 
IMATE- INaATe*lOOi 

KAITE Input tecwical factors 

«00 IF UKOT.Rpri) CO TO 110 

MRITS (iilOOO) COURSCiORATEiNi iNRATIi IMAOI n • I-li 10) • 



CCRAOl 1 1 
CSTIN( I ) 
7*0 C5FM ) 
APiOUP 
AAIQUP 
NFACIL 
OrAClL 
IFACIL 
FFACIL 



\ l«RAOtI > iI-U iIO) lAKATf iTSPOIO.TUENCHiPes^.ltCKATf I 

^ 2 EOTifO ^ 

1000 FOUNAT tM',itX,'COUft$C COST ANALYII S • , /, U , • IN»UT I • , /, JX , 

1 •^COURSE miMOCA - * i A4 , 25X , • 0 1 SC0UN7 RATI - •iF4.1f 

2 • FIRDENT' i/.lJIi'FUANNINC FERlOO - *tli,2lK, 
i 'INFLATION RATE - •,F4.l,t PIRCINT • , //, IX , 

4 'INPUT TeCHNICAL F ACTOR S t », /f >X ,' PLN« YR',' I Jt, 

9 ' 3 4 9 4 T I 1 S 

4 '10',/, JX,'tf»«0S-',10Mx,|4),//,Jir,»PUN6 Y« ' , » 11 ', 

T '12 IS U 19 14 17 II lf» , 

8 ■ 20'4/,3X, 'CRAOS-i , 10111,14), //,)X, 

• 'ATTrtiriON RATE tFCTJ - ',Fa*2,IS, 

1 'TInE STUO fOS OOUH IFCT) - 'tF).2t/fSXt 

4 'TRAININC LENGTH INKS) - ',F9.1,«X, 

0 'FCT EXTRA STUO POSITIONS - •,FS.2,/,SX, 

C 'RCCYCLS RATE IPCT) - ',FI.2,9X, 

0 '2QUIP/STU0 POS RATIO - ',F4.2) 

URITC rA,lQl01 ARCYTn,L0FEQ,HSCHOPl(N0DEV,INTSra,^IM0, 
) A^rS^OiUPOAri ,SQFTST*EVlM,I^TlN,&UPlPO,t*f TAN, 

SUFiTO,PURCMF,NSCJPO,0«PF,NICSTO 



1010 FORNAT 



'AVE. RfCYCLE TINC IMS)- 
•LIFi OF EQUIPNfMT 
'MEiKS SCHOOL QPERATfS - 
'COURSE OEV HUS/COiJRSE HA - 
'INSTR/STUO POS RATIO 
■FCT COURSe KEQUIRINC OIV.' 
MOHIN/STUO FOS RATIO 
'FC7 INITR MAT. NAINTAlNIIK 
'SQ F(I7/STU0ENT FOS. ^ ', 
'PC7 INtTR HAT. RNNC VA4.U4- 
•»0 FIIT/IMSTB. POI. - • , 
'SUPPLIfiS/STUOiNT POtlT10H> ',F9.1 
'SQ FEET/AOMiN POS. - •iF4.2 



,F4.1i«X, 

',12, /fix, , 
,F4.1,«X, 
',F9.1,/,1K, 
• ,P4.2,7X, 

• *F4.2,/, IX, 
',P4.},7X, 
',F1.2,/,1X, 
F4.2,IX, 

•,F1.2«/|>X, 



IX , 



/,1X, 



'SUPFLIEl/STUOENT 
'PUnCHASE POLICY 
'nISC/STUOENT POSITION 
'DEPRECIATION POLICY 
*MlSC/tTUOfNT 



■X, 



,F9.1«/,)K« 
tl lillK, 
.F9.1,/,]X, 
«11,11X, 
, F9.lt 



WftlT^ INPUT COST FACTOftS 



910 IF {.NOt.Rr>72> CO TO r^O 

WnlTC I4«100l) COURSE,ORAT£,N,INHAT|,<CO^NTII»,I«l,lO), 
» tCOPNTII I, 1-11,20) ,STU0SL,CIN0,STCS71,CIU#F,I7CI72, 

• i CNIS,INSTSL,FCTPCH,A0NSL,CFSQFT,10UNTC 
1001 FORMAT i'l',20)(,'COU(lSE COST ANALY 1 1 1 ' « /, 1 X , • | N^UT I ' , / 1 >R t 

1 'COURSE NUMQKR - ' • A4 , 29X , ♦ 01 ICOUNT lATE - 'rf^.lf 



ERIC 

hfiiinniirnrmaaia 



IV-30 



TAEG- REPORT NO. T2-3 



04/23/79 
CAKO NO 



INPUT LISTING 



AUTOflQN CHAMT SET - 



1*2 
7i3 
Tl« 
715 
7I» 
7iT 
7II 
7lf 
7*0 
7»l 
7f2 
7*3 
7«* 
7»5 
7«* 
7*7 
7«l 
7** 
•00 
•01 

■ 02 

■ 03 

■ 04 
•09 

• 6h 
•07 
■0^ 
■0« 

■ 10 

• 11 

■ 12 

■ 13 

■ 14 

■ 19 

■ 14 

■ 17 

■ !■ 

■ 19 
■20 

■ 21 

■ 22 

■ 23 

■ 24' 

■ 25 

■ 24 

■ 27 

■ 2^ 

■ 2« 

■ 30 

■ 31 

■ 32 

■ 33 

■ 34 

■ 39 

■ 34 

■ 37 

■ 3t 

■ 39 
■40 

■ 41 

■ 42 
■43 

■ 44 
■49 

■ 44 

■ 47 

■ 4* 

■ 49 
■90 

■ 91 

■ 92. 
«93 
■94 

■ 99 

■ 94 

■ 97 

■ 9« 

■ 99 
840 

■ 41 

■ 42 
■43 

■ 44 

■ 49 
144 
• 47 

■ 41 

■ 4« 

■ 70 

■ 71 

■ 72 

■ 73 

■ 74 

■ 79 

■ 74 

■ 77 

■ ?■ 

■ 79 

■ ■0 

■ ■1 

■ 12 
013 

■ ■4 



1002 FOKMAT (M 



COAOCCI AOC.CFAOCtUNAOC.UOAOC.UIAOC.UFAOC.PNAOC. 
^OAOCtPiAOCtfFAOC 
•2^Kt'C0URSE COST ANALY S I S • , / , 1 X , ' TNPUT I • , / . 3X . 



1 'COURSE NUHIER • ' i A4 , 29X i • 0 1 SCOUNT MATE - •|F4.1» 

2 • PERCENT*, /,3X, 'PLANNING PERIOO - •,I2,27X, 

3 'INFLATION RATE - ».F4,l,' PERCE NT • . / / , 1 X . 

* 'OUTPUT COST RESULTSl'.lOX, 

4 •NOM-OISC OlSCOUNTEO NON-OISC 0 1 SCOUNTKD • i / , 3 IX , 
9 'CONSTANT I CONSTANT I CURRENT t CURRENT ■ • , / . SX . 

* 'STUDENT SALARIES', 10I,4(2X,F10. 01, /,3X, 
7 'STUOENT TRAVEL', 12X,*I2X,F10. 01, /,3X, 

■ 'INSTRUCTOR SALARY' ,»X,%f2X,PlO.O) ,/,3X, 

* 'AOHINISTRITIVE SALARY', 9X,4I2JI,F10. 01, /,3X, 

1 'EWIP«ENT»,l7x,4UX,Fll.W,/,3X, 

* 'FACILITIES' •14X,4( IX, Fn.0),/»3X, 

■ ' INSTRUCTIONAL M A T E R I AL S • , 3 X , 4 ( 2X , F 1 0. 0 | , / , 3X , 
C 'SUPPLIES' ,1«X, 4 (2X.F 10.01 ,/,»X, 

0 'MISCELLANEOUS' •13X*4(2X,F10.0),//i3X, 

E 'TOTAL OPERATIONAL' ,9X,4 I lX,Fn.O) ,//,3X, 

F 'COST PER CRAOUATE' ,RX,4(4X,F«.2),/,3X« 

C 'COST PER STUOENT I NPUT ' , 3X ,4 ( 4X , F«, 2 I , / • 3X , 

^ H 'COST PER STUOENT .f0SITI0N',4(4X,F«,2M 

C WRITE ANNUAL COST RESULTS 

C 

«90 IF I.NOT.RPT*) GO TO 940 

HAITE Ik, 1004) COURSE, ORATE. NilNRAtE.tOlGITI I),]. 

t <CSTUOf,I I , 1-1.9) . (CTRAVt n , I -1 • 9) , 

2 (CINSTR-( I ) , I-) ,9 I , ICAOMlNl I ) , 1-1 ,9) , 

3 <CEQUIP(I),I-1,9),(CFACILII)*I-1,9), 

4 tClMOMTI I ) ICSUPPLt I ) . 1-1 .9), 

5 <C«ISCU » .1-1,9) ,iCOPTOTm , 1-1,91 , 

■ ICGRAO( I ),I-I,9),(CSTIN(I),T-U9), 
7 ICSPII), 1-1,9) 

IF fN.LE.9) GO TO 940 

HRI TE 14, 1004) COURSE, OR ATE ,N, INRATE , I OIG I T ( 1) , t -k , 1 0) • 

1 (CSTUOi n . I>*4. 10) .ICTRAVt t ) . 1-4. lOt, 

2 (CINSTRI I ) .I-k.lO) . (CAOMlNd ). 1-4.10). 

3 ICEQUIPd ). I-k.lO). (CFACILd) .1-4. 10). 

4 (CIMOMTI I ). I-4.10).(CSUPPL( t ) . 1-4. 10) . 
9 ICMISC( I I . l-ft. 10) , (COPTOTt I ) . I-*.10) . 

■ <CGRAD(I).I-4.10).(CSTINI1).I-4.I0). 
7 tCSPd ) .1-4. 10) 

IF (N.LE.IO) GO TO 940 

MRITE lft.1004) COURSE. ORATE. N. INRATE. OIGIT(I). 1-11.19). 

1 (CSTUO( n . 1-11. 19) , rqTRAVI I ) . I-Il. 19) . 

2 iCINSTRI I ). 1-1,1.19) . (CAOMINI I ) .1-11. 19), 

3 (CEQUIPI I I .1-11.19) . tCFACiLt I ) .1-11. 19). 

4 ICIMOMTU ) . l-U . 191 . ICSUPPLt I ) . I-U . 19) . 

5 ICMISCI f) . I - II .19) . (COP TOT d ) , l-U . 19) . 
4 ICCRAOl I)il-tI.t9).ICSTlNd).}-lI.I9). 
7 (CSPd ) . I-U . 19) 

IF IN.LE, 19) GO TO «kO 

WRITE 14. 1004) COURSE. ORATE. N. INRATE. (OIGIT(I), I-U, 20), 

1 I CSTUOI I ) . 1-14.20) . (CTRAV I tt ,J-U.20). . 

2 ICINSTRI I ) . I-I*.20) . (CADHINI I ). 1-14.20) . 

3 (CEQUIPd ) .1-14.20) . tCFAClL( n .1-14.20). 

4 ICINONTI I ). I -14.20) . iCSUPPL d ) . t-l4.20| . 

5 (CMISCI I ) .) -14.20) . I COP TOT d ) . I-U.20). 
4 ICGRAOd ) . I - 14.20) . ICSTINd t . I-I4.20) . 
7 (CSPd ).I-I4.20) 

1004 FORHAT I ' I ' . 2 ■ X . » COUR S E COST AN AL YS I S ' . / . 1 X . ' I NPUT I ' . / . 3X . 

1 'COURSE NUMSER - ' . A4 . 29X . » 0 1 SCOUNT RATE - '.F«,l. 

2 ' PERCENT'. /.3X. 'PLANNING FERIOO - '.I2.27X. 



3 


' INFLATION RATE - ' .F4. Ii ' 


PERCENT ' .//. Ix 






4 


'OUTPUT COST RESULTSI JNON 


-OISC 


CONSTANT 


II'.// 


32X 


5 


'YEAR '.I2i' YEAR *.(2.' 


YEAR 


' . 12. • 


YEAR 


. 12 


4 


' YEAR • .12.//. 3X. 








7 


'STUOENT SALARIES 


'.9( IX 


.F».0) ./ 


3X. 




0 


•STUOENT TRAVEL 


• .9 dX 


»F»,0) ./ 


3X. 






•INSTRUCIOR SALARY 


♦,9I IX 


.F9.0) ./ 


^X. 
3X* 




A 


' AOMINISTRAT IVE SALARY 


',9dx 


,F»,0)',/ 




B 


'EQUIPMENT 


' .9 dX 


.F9,0) ./ 


3X. 




C 


'FACILIT lES 


' .91 IX 


.F».0) ./ 


3X. 




0 


'INSTRUCTIONAL MATERIALS 


'.91 IX 


•F»,0) ./ 


3X. 




E 


'SUPPLIES 


' .9 dX 


.F».0) ./ 


3X. 




F 


'MISCELLANEOUS 


• .9dX 


F»,OI ./ 


3X. 




G 


•TOTAL OPERATIONAL 


• , 9dX 


F»,0) ./ 


3X. 




H 


'COST PEH CRAOUATE 


' .9 1 IX 


F»,0) ./ 


3X. 




\ 


'COST PER STUOENT INPUT 


' .9 dX 


F»,0) ,/ 


3X. 




J 


'COST PER STUOENT POSITION 


' .9 dX 


F9.0) ) 







C MRITE KISSING STATEMENT Nu4«ERS 

C 

»40 IF I.NOT.RPTT) GO TO »90 
IF (K.LE.d GO TO 490 
WRITE (fc.9000) 
K - K-1 

WRI TE (4.9001 ) ( (MATA I I ) .MAT«( I ) ) , I-l .K ) 

9000 FORMAT Id'.' 

9001 fORMAT dX. 'MISSING UATA COOE - 
990 IF (.NOT. LAST) GO TO 10 

CONTINUE 
ENO 

SU*RnUT{NE MISSNG <NUM«.|CaOE) 
DIMENSION MATA ( 100) .HATBI 100) 
COMMON/NOGOF/NQGO.K.MATA.MATIt 
IF dCOOE.EQ.O) NOGO-1 
MATAIK) - NUHR 
MATKKI - ICOOE 



Ml SSING OATA ANALYSIS' ) 
I4,^X.'CRIT COOE - '.14) 



RtruRN 
ENO 



ERIC 
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TAEG REPORT NO. 12-3 



0^/21/79 . AUrOPLOM CHART SET 

t 

CHMX TITLE - IMTROOUCTORV COMMENTS 



•••••••••••••••••••••••••••«•••••••••••••••♦♦••••••••••«•••••••••• 

• moCRAH - TIIAININ6 COST MOOf L • 

• TECEP COST NOOiL AOAPTATIOMS POM ETAM IV L* OUfFV • 

• OATf - 4/11/79 • 



i»KOGAAM INITIALIZATION 



ERIC - ]9-4r 



TAEG REPORT NO. 12-3 



i. 



CHMT tlTLC ' raOCIOURES 



AUTOFLM CHART SIT - 



/ 10 
17.14 >l 



01 

AASIfi - 0.0 I 
AAOl • 0.0 

TiNsm - o«a. 

TAOMIN - 0.0 

JlQUlP ' 0.0 

TFACIL - 0.0 

I 
I 

(000099) I 02 



TKOKNT - 0.0 

TSUFFL - 0.0 

TNISC - 0.0 

TMQUF - 0.0 

TACQUP - 0.0 

I 

(000041) I 

TPCQUP - 0.0 
AAEOUP - 0.0. 

PURCHF - nnnn 
o(E^F - nnnnn 

RVCQ - 0.0 

NSTUO - 0.0 

] 
I 

(oooo4Tr I 

OSTUO - 0.0' 
ItTUO - 0.0 
FSTUO o 0.0 
HfRAV - 0.0 
OTRAV - 0.0 
ITRAV - 0.0 

I 



— - • 



(000091) I 

FTRAV - .0.0 

NINSTR - 0.0 

OINSTR - 0.0 

I iNSfR - 0.0 

FINSTR - 0.0 

NAOMIN - 0.0 

— • 

I 
I 

1000099) I 0* 
• • 

0/lOMlN - 0.0 I 

lAOMlH - 0.0 
fAOMiN - 0.0 
NFACIL - 0.0 
OCACIL - 0.0 
IFACIL - 0.0 
I 



I 



>• 

(000049) I 



FFACIL - 0.0 

NIUPFL - 0.0 

OSUPFL - 0.0 

ISUPFL - 0.0 

FSUFFL - 0.0 

NMISC - 0.0 



I 
I 

(OOOOTl) i 



OMISC 




0 


0- 


IMISC 




0 


0 ' 


FMISC 




0. 


0 


NEjOUIF 




0 


0 


OCQUIF 




0 


0 


(EQUIP 




0. 


0 



on 

• — - — • 



(OOOOTT) J 

FEQUlF - 0.0 

N[MO - 0.0 

OlMO - 0.0 

(INO - 0.0 

FiNO - 0^ 

OAOl - 0.0 



• 



(00001)1 I 

SAOl - 0.0 
NOCO - 0 
ICRAO - 0 
IFSF - 0 
JAOR - 0 

I ARATE - 

• - — — — 

I 
I 

(OOOOIfl I 

• _ — 

RCRATE - ninnn, 

ARCYTM - 
MSCHlF - 
TSFOSO - 



I 
I 

tOCOO«)) I 
FESF - 
INTSFO - 
AMTSFO - 
iQTSFO - 



OT 



01 



— >• 

(oowiyi I iJ 
• — — — - — • 

LOFEQ - 
SQFTST - 
IQFTIN - 

SQFTAM - nnnnn. 







— — • 



IHOOEV - 
PUlMO - 
UFOATE - 
SUPSPO - 



SUFSTO - «M««. 
HSCSFO - 

HSCSTO - nnnnn, 

STUOSL - 



STCSTl - nnnnn* 

STCST2 - ««M«. 

INSTSL - nnnnn, 

AOMSL - 



U 

I 



EQUNTC - t««««. 
FCTFCH J" 
CFSQFT - 

CIMO - nnnnn* 



EVlM - 
CSUFF - 

CMIS - 
ORATE - ««M«. 



I 
I 

I00012U I 1« 

- — 

INRATE - 

N - nnnnn, 

TLENGH - M«««. 
K - 1 



(000129) I NOTE 20 

• • 

• l(6(N 00 LOOF • 

• 20 I - li '20 • 

I 



I 21 
— . • 

CSTUO(I) - 0.0 

CTRAVd) • 0.0 

CINSTR(I) - 0.0 

CAOMIN( 1 ) - 0.0 
• — - — • 



(000130) I 
• -^--.^-^ . 

CEQUIf ( I ) - 0.0 

CFACIL(I) - 0.0 

CIMOMTI I ) - 0.0 

CSUFFL(I) - 0.0 

• • 



I CMISC(I) - 0.0 

I 

I C0FT0T( I ) - 0.0 

1 

I CCRAOII) - 0.0 

I 

1 CST1N( I ) - 0.0 



I 



CSF(I) - 0.0 
FOEF(I) - 0.0 
AEQUIFI 1 ) - 0.0 
STUO( I ) - 0.0 
AOII I) - 0.0 



I FSF( I ) - 0.0 

I 

I R(NSTR(1) - 0.0 

I 

I RA0MIN( I ) - 0.0 

I 

I REQUIF( I) - 0.0 



I 



FEQUIF(I) - 0.0 

AEQUIF( I ) - 0.0 

RFACIL( I ) « 0.0 

lMDMNT( I ) - 0.0 



I RSUFFK I ) - 0.0 

1 

I RMiSCIl) - 0.0 

I 

I AOIX(I) - 0.0 

I 

I GRAOII) - 



(000)99) I 21 

— - — - — - — • 

I COFMT(I) - I 
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TAEG REPORT NO. 12-3 



ChMT TITLI " MOCIOlMfS 



AUWLOK CNAUT MT - 



ENO OF □□ 
iOGFT 



I 

2 
21 



02 

/ / 

/ REAO 'man ocv / 
/ ? / 

/ VU PONNAT / 

/ TCCMf / 

I 



/ HEAQ man oev / 

/ ? / 

/ VIA fONNAT / 
/ CDSTF / 

1 



/ NIAO PNON OIV / 

/ » / 

/ VIA PMNAT / 



( 
I 



OS 



/ NCAO fnan qev / 

/ 5 / 

/ VIA PONNAT / 
/ 4000 / 

/ INTO THE |.ISr / 

I 
I 

I000U«) i HQM 0& 



LIST » RPTli 
IIPT2( nPTlt KPT4, 
Rprt* RPTft* RPTT, 
LAST 



I 



N .Nf. ««««« 



01.0t»— >• 

29 I 



01 



/ HBAO PROM OEV / 
/ 10 / 

/ VIA PO«»WT / 
/ *020 / 

/ INTO r»>f LIST / 

I 

I' 

(OOOlTf) I NOTE 10 



LIST » COIMSX. 

NRi ARATEXi 
TLIfMHf RCRATXi 
ARCVTX. fMMAOIDi 
lt«IIA0(2li 
MMAOdI I 
AQtXtlit AOtXII). 
AOtX(l) 



I 
I 

J 

t000l«79) • II 



• • VES 

• INO OP DATA7 • 



TRUE • 

* • CQUnSX .NE. 

• CQURif 



1 



11 



I 

(oooir«i I 

I N - NX I 

I I 

I AKATE - ARATIX t 

I I 

I TLCICGH - TLENGX I 

I I 

I RCRATE " RCRATX 1 

I 
I 

tOOOltS) I 1« 
• — - — — 

ARCYTX I 



I ARCVTN 



(OOOlt^l I NOTE 19 

• IfGIN 00 LOOP • 

• 21 1 - li N • 



• ...» i 


1 




1 12 1 




H 


iPAtSZ .S.I 


OS. IT >l 




1 IS 1 


MiSfM 


H 


1 . 11 . 1 


21 1 


1* 


1 1. 1 


(40. 1) 


H 


1 1 






t to 1 




H 


1 SO 1 - 


1 GRAO(I) - 


h 


1 11 1 




H 


1 1 
i 1 


1 MMAOin 


1 




1 




1 1 
t(«AI TPf NOOffL OATA | 


1 
1 






1 
1 




1 1 
1 1 








1 












/ «.0t 





C0001I9) • 



END OP 00 
LOOPT 



S 
14 



REAO MISSING OATA 
ANALYSIS ANO OEPAULT 
ROUTINES 



— OS.OT*- 

30 



TRUE 



IPALSE 

I 
I 
I 
I 
I 

i 



20 



12 I H 

1) i NISSNG H 

I. I (10,11 H 

10 I H 

111 H 



•ARATl .GE. 

0. .ANO. 
•ARATE .LT. 

• 1. • 



(0001411 I 
! ARATE - 



TRUE 

I 
I 
I 



01 . 

"so 



22 



ERIC 



lV-34 



y > 



\ 

'A TAEG REPORT NO. 12-3 

\ 

\ 

<*^'2J/75 ^ ^ *UTOFLO»t CHART SET - 

CHMT TITLE - Wlo'cepUHES 



TRUE •Tj.EfiCH .CT.» 

• • 0. VANO. 

^•TL€N6h .LT.'* 
• 209. • 



« 0*.07*— >| 



12 1 

Is I 

10 I 

11 1 



(50rO) \ 



TRUe •HCRATe .CE.» \ 

• 0. .ANO. • \ 

•RCRATE .LT.* \ 



40001«ll I 04 

• • 

1 RCKATE • .S*AKATE | 



12 I 
|] I 
U 1 

10 I 

11 I 



Ml SSNC 

uo, n 



•ARCYTM .GE.* 
0. .ANO. 

•ARCYTM .LT.» 
• 200. • 



1000201 > • 



• KCRATE .EQ. 



11 . 



I 000202) • 



• TLENCH .LT. 



12 I 
\i I 
I. I 
10 I 
fl I 



HI SSNC 
( 70*01 



AHCY7M • 0.0 



(00020*) I 
• - 

12 I 

13 I HI SSNC 

I . I nOi 1 ) 
10 I 

II I 

• 

I 



TRUE •WSCHO^ .CT.* 

• 0. .ANO. • 

•MSCMOP .LE.* 
• 52. * 



(000201) I 14 
* « • 

I WSCHOP - 50.0 I 
• — • 

I 
I 

100020*) I 15 

• • 

12 t H 
|3 I NISSNC H 

I. I l«0«l) H 

10 I H 

II I H 
• • 

I 

_>, 

I 

100 • t4 
• • 

• • 

TRUE 'TSfOSO .CE.* 

• 0. .ANO. • 

•TS^OSO .LT.* 

• I. • 



iFALSt 

I 

I 

I 

I 
I 

{000?l 1)1 17 

— V- * 

I TSfOSO - 0.0 I 
• • 

I 
I 

(0002121 1 18 
• • 

12 I H 

13 t MISSNC H 
I. I (100«lt H 
10 I ' H 

II I H 
• • 

I 

>l 

I 

110 • 1* 



PES^ .CE. * 
. .ANO. rtip 
.LE. 1. * 




IV-35 



TAEG REPORT NO. 12-3 



CHAlir TITLf - HtOCIDUMIS 



0*.l» >• 

(0002L4) I 



<0002I9) I 

• 

12 t 

It I MIIIM 

U I (LlOfU 

10 I 

11 I 



I 

0*.l» >| 



« 0. •AHO*- 

• I. • 



I 



I 

, I 
I 

I 

I00021T1 t 



I 



04 

I 



loooiia^ 



12 I 

11 I 

N I 

|0 1 , H 

II I N 



MISINC 

120* II 



♦ 0. .ANO. 

•AMTSPO .LE. 
• I. • 



12 I 

13 I 
U I 



MtSSNC 

(1I0»L) 



•IQTtPO .M. 

0. .AHD. 
•iOTS^O .Li. 

• I. • 



IPALU 



5 
LA 



(000224) • LI 



•LOrCO .NE. • TRUE 

• .ano. • 

•LOfEQ .NE. • 
• 0 • 



-05.L0»— >| 

I 



12 I 

13 I - MllfKC 
I. I (L^O.Ot 

10 I 
IL I 

• - — — 

I 



TRUC •PURCHP .BE.* 

• I .AND _ 

•rUiCHF .LT.» 
• LO 



(0002S0) } L7 

• — • 

I PURCHF m I I 



(0002SLt I 



12 I 

IS I MIXING 
I. I (140, LI 

10 I 

U I 

c — - — — . 
I 



TRUE •OtPf .CE. I* 

• .ANO. OEPF 

• .LT» 10 • 



10002231. • 10 1 

* * 1 

* * 1 
•EOUNTC .NE.* TMUE 1 

• «t«M. .AND. • ♦ 

•EOUNTC .NE.* A 

• 0. • 1 

* * 1 

• 1 


1 • 

1 , IFALSE 

1 1 
1 1 
1 1 
1 1 
1 1 
t (0002SSI 1 


20 1 


IFALSE 1 

1 1 


1 1 OEFF - I 


' 1 1 


1 1 
1 I 
1 1 
1 1 


1 1 
1 1- 
t {0002341 1 


2L 1 



12 I 

13 I MISSNC 
I. I (ITOdl 
10 I 

IL 1 



I 



IFALSE 



I 

(0002251 I 12 

I EQTSFO - 0.0 t 

I 
I 

(00022AI I LI 

12 I *" H 

|3 I MISSNG H 

I* I (140, LI H 

10 } h' 

U I H 

• • 

I 



•LDFEQ .CE. • TRUE 
• I .AND. LOFEQ 
• .LE* 90 • I 



(000234) • 2) 

• • TRUE 
• EQTSFO .NE. 

• 0. • I 



IFALSE . 4 . 
I . 01 . 



\ 



(0002171 • 24 



•EOUNTC .NE.* 

.AND. 
•EQUNTC .NE.« 
• 0. • 



IFALSE 



10002311 I 
I LOFEQ 



(00021fl I 



25 

I 



12 1 

19 I MISSNG 
I* I ILtO,LI 

10 t 
IL I 

I 



TAEG REPORT NO. 12-3 



CHAiir ririE - mqceoures 



AUTOPLOM CHART SST 



05.23*— >| 



12 I 
U I 
I. I 

10 I 

11 1 



MISSNG 



TRUE •SOFTST .GE.* 
— — • 0. .ANO. 

•SQFfST ,11. • 
• 1000. • 



13 I 



10 I 

11 I 



HISSNC 
(200, 1) 



TRUE 'SOFTIN .CE.« 

• * 0. « ANO« 

•SQFTiN .LT.« 
• 1000 « • 



IFALSE 



SQFTIN - 0.0 



12 I 
U I 



10 I 
U I 



1 210, n 



•SQFTAH .GE.* 
0. .AND. 

•SQFTAH ,LT.* 
• 1000. • 



1 

(0002A3t 1 


03 


1 12 1 


♦ 






1 U 1 


, HISSNC 


1 SQFTST - 0.0 


1 


1 U 1 


(220, n 


1 




1 10 t 
1 U 1 





>• 

(000290) I 



•IHOOEV .GE.* 
0. .ANO. 

•IHDDEV .LT,* 
• 1000. 



(000292) • 12 



•PUIHD .CT. 

0. .ANO« 
•PU)ND .LE. 

• I. • 



(000293) • 



•upOATE .cr. 

0. .ANO. 
•UPDATE .LE. 



(00029A) f M 

• ' 1 

' I IHOpEV - 0*0 I 



12 I 

13 t 
I. I 
10 I 
U I 



HISSNC 
(230, 1) 



06. 12* — >• 

240 I 



♦ Ok. !♦•—>• 

'2kO I 



i2 I 

|3 I HISSNC 

I. I (230,0) 

10 I 

U I 



I 



•PUIHO .CT. 

0. .AND. 
•PUIHD .LE. 

• I. • 



IFALSE 



(000299) • 



IHOOEV .HE. 
» 0. • 



(0002fcDt • 20 



•UPOATE .CT.* TRUE 

0. .ANO. 
•UPOATE .LE.* I 
• 1. • I 
• • I 
• . * • . 

IFALSE . k . 

I . ?A . 



(0002^1) i 
I PUIHD - l.O 



(000242) I 



12 I 

I 3 t HISSNC 

I. I (290,1) 

10 I 

a I 



t 



12 I 

13 I HISSNG 
I. I (290,0) 



I 



•UPDATE .GE.* 
0. .AND. 

•UPDATE .LE.* 
• I. • 



TRUE 
•-♦ 

I 
I 
I 



(00026*) • 2* 



• IHOOEV .NE. 



TRU,E 

I 
1 
I 



(0002*7) • 



•PUIHD .GT, 
0. .ANO. 

•PUIHO .LT. 
♦1. • 



(0002*1) i 

— 

I UPDATE • 0.0 



T 

Ot 



1/ 



I0D02*«) I 



121 
13 I 
I. I 
10 I 
U I 



HISSNC 
(2T0,n 



0 

ERJC 



IV- 37 



TAE6 REPORT NO. 12- 3 



CHAUT Tirte - f«ocioufiet 



AUmiM CHAftT MT " 



*I I 

13 I 



10 

II 



Ok 



NIttNC 
(170(0) 



TMUf •tU^S^O .Of.* 

• 0. .AND. 

•tU^l^O .LT.« 
• 1000. • 



(D002T1) I 01 

• • 

] IMfifQ m 0.0 I 



t 00027^1 I 
1 2 I 

t3 I mSSNC 
I. I (2fO>l) 

10 I 

ILL - _ 
I 

♦ >j 



• 0. .AND. 

• Ui^STO .LT.* 
• kOOO. • 



(0002741 I 



J000277» 1 

12 i 

tl t 

1. 1 

10 I 

11 I 



0.0 j 



•NlCSrO .GE.* 
0. .AND. 

•^SCSrO .LT.* 
« 1000. • 



(0002791 i 
' 1 NlCt^O « 



(0002101 I 10 

12 I ***H 

IS I MISSM H 

I. i (SkOtU ^ H 

10 I H 

III H 



07.11 >• 

(OOOXtVI I 



12 i 
IS I 
I. I 

10 I 

11 I 



MIttNC 
(J*0,l| 



->l 



•MSCSTO .GI. 

0. .AMD. 
•HICITO .LT. 

• kOOO. «* 



NSCSTO « 0.0 



t0002ISI I 

12 I 

IS I 

I. I 

10 1 

I I I 



NISSNG 
( 32 0« 1 ) 



•STUOSL .Gi.« TAUC 
0. ,ANO. • 

•STUOSL .LT.* 
•fOOOO. • 



I STUOSL « 0.0 I 



(0002 It) I 
• — — 

12 I 

Is t MISSNG 

U 1 (sso,i) 

10 I 

H I 



•STCSTk .CC* TKUC 
0. .ANO. • 

•STCSTl .L7.« 
•lOOOO. * 



I0002tt) I ^11 

I STCSTl - 0.0 I 



•STCST2 .Gt.* 
0. .ANO. 

•STC$T2 .LT.» 
•kOOOO. • 



I 
I 

I \ 

t0002«kl 1 ^2k 

I STCST2 • 0.0 1 I 

• J., 

. ' I 
(b002«2) I / ' 

12 I ' 
I S I MISSNG \ H 

I. I (SSOil) \ H 
10 1 ?H 
Ik I fH 

I \ 

|< 

SAO • 2S \ 



•INSTSL .G(.« tftUe 
0. .AND. 



•INSTSL .LT.« 
•»0000. * 



(0002U) j 



1% 



INSTSL « 0.0 



(0002*51 I 



2*1 



I? I H 

IS I NltSNG H3 

t. I (StOit) H'. 

10 I 



•AOMSL .Gi. • 
0. .AND. 

•ADMSL .tT. • 
• »0000. • 



THUJ 



AONSL - 0.0 I 



07.27 >• 

(0002911 I 



12 I 

IS I - MISSNG 

I. I (S70«U 

10 I 

11 I 



— >i 



•eOUNTC .G{.« TAUC 

0. .ANO. ' 
•EQUNTC .NC.« I 
•99<m. • I 
• • * I 
• .... 
IFALSC . ■ . 

I . 02 . 



(00030C) • 



• • THUi 
• tQTS^O .NE. 

• 0. • I 



I FALSE . I 
I . Ok 



(o(rosoki I 



I lOUMTC - 0.0 



1000)02) I 



12 I 

IS 1 MISSNG 

!• I (SIO,k) 

10 t 

Ik I 



I 



ERJC 



IV-3B 



TAEG REPORT NO. 12-3 



04/23/79 
CHAKT TITLE - PKOCEOUKES 



AUTOFLOW CHAKT SET - 



I 



12 I 

19 I HtSSNG 

I. I mo.o) 

10 I 

11 I 



I 



TKUE •PCTPCH .G£.* 
— — • 0. .*M0. 

•fCT^CH .Lt.» 



IFALSE 

I. 
I 
I 
I 
I 

(00030*1 I 03 

• • 

I FCTPCH • 0.0 I 
• — • 

I 
1 

(000307) I 04 
• • 

U I H 
I) I HISSNG H 
I. I (400.11 H 
10 I H 
111 H 
« — • 

I 

>l 

I 

MO * Ob 



TIlUE •CPSQFT .GC* 

• 0. .*N0. 

•C^SOFT .N£.# 



(FALSE 
I 

I 

1 

(0003041 i 04 
« « 

I CFSQFT • 0.0 j 
• — • 

I 

.> . I ' 
(000310) 1 j 07 
• ^ • 

12 I : M 

I) I HISSNG H 

1. I (410*1) H 

10 I H 

II I ^ H 
• ft 

I 

.........>, 

420 • 01 

• CIMO .GE. ' • THUE 

• 0. .ANO. CIMO • 

♦.LT. 50000.* 



C(HO • 0.0 



>• 

(000313) I 



Ofl.iB >• 

(000322) I 



12 I 

I) I NISSX6 

I. I (420fli 

10 I 

11 I 



I 



• EVIH .GE. • TRUE 
• 0. .ANO. EVIM • 

• «LT. 1« • 



IFALSE 



I 

^ (000319) I 12 
• • 

^ I EVIH • 0.0 I 
• • 

I 
I 

1000314) 1 13 
« « 

12 I H 

13 I . NISSNO H 
I. I (430.1) H 

10 I H 

11 1 H 
« # 

K 

4*0 • 14 

• • 

•CSUFF .G£. • TKUE 

• 0. .ANO. • 

•CSUFF .NE. • 



CSUFF n 0.0 



(000314) 



12 I 

13 I HISSNG 
t. I (440>1) 

10 I 

11 I 



I 

l<- 
I 



• CHIS .GE. • TKUE 

• 0. .ANO. CHIS • 

•.NE. 4^f4*.* 



I 00032 U I 



'12 I 

I) I HISSNG 

I. I (490tll 

10 I 

11 1 



I 

->l 
I 



•OKATE .GE. • 

0. .ANO. 
•OKATE .LT. • 



IFALSE 

I 
I 
I 
I 

(000924) I 21 
• •-• 

I OKATE • 0.10 I 

I 
I 

(000329) I 22 
• — • 

12 I H 

13 I HISSNG H 
I. I (440fl) H 
10 I H 

II I H 
« « 

I 

I < _ 

I 

470 • ,23 
• • 
• • 
•INKATE .GE.* TKUE 

* 0. .ANO. 

•INKATE .LT.* 
•1.4 



IFALSE 
I 



INKATE • 0.01 



12 I 

I 3 I HISSNG 
I. I (470,11 
10 I 

II I 



I 



4a(] • 24 
• • 

•GKAOIN)* 

• .GE. 0. • mix 

.ANO. GftAO(N) • 

♦.Nt. 4444^.« 



-T- ♦ 



01.24 >• . 

(00(^330) I \ 

• — -I v~ 

12 1; 

13 I I HISSNG 
I.I. (4to,or 

10 I 

III* 



I 

->l 



\ 



4<i0 I NOTE > 21 
0 m, •••••• 

COflHT(N) .GE. 0. \ • 
.jjNO. COFHT(N) 



.NE. 44444. 



I 



(000331)' • 



SEE NOTE 
ABOVE 



(0003321 I NOTE 30 



•EGIN 00 LOOF 
900 I • 1* N 



Oi.32 >l 

900 I 



I COFHTII I - 0.0 



. (0003 33) • 



• NO 

ENO OF 00 •-♦ 

L00F7 • t 

• • I 



(000334) 



\2 I 

13 I HISSNG 

I. I (490.1) 

10 I 

11 I 



1 



• NOGO .1:0. 1 •- 



17 
II 



ERIC 



IV-39 



TAEG REPORT NO. 12-3 



CHMT TtTLt - MI0C1DUII1$ 



0#.34 >• 

I 

(00035*) « 01 
• « 

• N .6f. 20 



10001171 I 02 

I iNoexs - N * 1 I 

I 
I 

lOOOIlIt I HOTE OS 

• ••••••••«• 

• leClN 00 LOOP • 

• 919 I - IN0EX9, • 

• 20 • 

I 

|< 

t 0^ 

• • 

I COPMT n > - 0.0 I 

• • 

I 
I 

^519 I 09 

I CIIAOtI ) - 0 I 
• • 

I 
I 
I 

(0001^) • 06 



KdO OF DO 



IVCS 



1 

>l . 

910 I NOTI 07 

• CONTIWJC • 

I 
I 
1 

NOOEL EQUATIONS - 
TECHNICAL FACTORS 

I 
1 

(0001^5) ) 01 



I UIMOTH - I 

I TLEWHAIMOOEV* I 

I FU1M0*^0.0 i 

I I 

I CUIMO - I 

I UIMOTM*CiMO I 

I I 

I «VIM • CUlMO«EVtM I 



ID00I4I) I »:OTE ( 

• ••••••••• 

• BiCIN DO LOOF 

• flO I • 1» N 



I 



— >• 

1 



• -5 

STUOl 1 i - I 
• NAOtn/.d.O - I 
MKATC) I 

I 

IMAO - IMAO * I 
MAOItl I 

I 

AASIN - AASIN * I 

STUO(l) I 
• • 



STUDMMfJ) - I 
(TLE#«««$TU0( n 1 I 
•d.O - .5«ANATEI I 

I 

snMMCd) - I 

RCRATE«5TU01 I ) ' | 
•AMCVTM I 

I 
I 

I000S94I I 12 

I Aoft( n - I 

t (STUONMIl) * \ 

I SMWftMC( 1 ) )/MSCHOF | 

I I 

I OAOl - OAOt * I 

I (AOKl) < I 

I Aoixini I 

I I 

I SAOt - SAOl * I 
I AOtdt I 



AAOB - AAOt * 
AOtd > 

FIFd I - 
I (AOId t/d.O • 
TSFOSO) )«d.O * 
WIFIl 

IFIF n IFSF ♦ 
■ PSFdl 



(OOOSAO) I U 

1 RINSTIId) - I 

I FSF(U«INTSFG | 

1 I 

I TINSTR - TINSTR * I 



RINSTIId ) 



t RADHINd) - 
I FSFd )*AMTSfO 



(000)«)t 1 19 
I TAOniN - TAtmiN * t 



RAOMIN( I ) 



REQUIFd ) - 
FSFd )*EOTSFO 



I TEOUtF - TSOUIF * I 



(0001*4) I I* 

I RFAC2L(n - I 

I FfFd)«(ieFTiT * I 

1 |NTtFO«IOf TIN * I 

I ANriM«ft«?TAM) I 

I I 

I TFACIL - TFACIL ♦ | 



.<000)*l) t 17 

I iPiom<Td) - 1 

I U1WTN*UF0ATE I 
I I 
I TNOttNT • TNONNT * | 



I RSUFFLd) >^ 
I FSFd)«$UFSFO * 
I STUOd )«SUFSTO 



10001711 I 



I T5UFFL ■ TSUFFL ♦ I 

I RSUFFLd I I 

I I 

I RNISCd) - I 

I FSFd >«NSCSFO ♦ 1 

I STUOd l«NSCSTO I 

I I 

t JAOl - JAOB * I 

I AOIXd) I 



I 



I TNISC - TNISC * I 
I -RNIS€(| ) I 



(00017^) • 20 



END OF 00 
LOOF? 



(000179) • 21 



FALSE • 
♦ • OAOl .LT. 0. 



(000179) \ 

• — - — 

I OAOB • OAOO«I-1. 



I I AO* - 

I ((OAOI/IAOO) * 

I .f)«100 



rv-40 • . 



TAEfi REPORT' NO. 12-3 



0^/23/7S 
CHART TITLE - PROCCOURES 



AUTOFLQU CHART SET 



0*.23 >• 

I 

(0003TT) • 01 



FALSE •lAOl .CT. 9* 

• .ANO. J AOS 

• .NE. 0 * • 



I TRUE 

I 
I 
I 
I 
I 

1000377) I 



12 I 

13 I 
I. 1 
|0 I 
II I 



MISSNC 
(920, 1 AO!) 



>| 



AASIN - AASIN/N 

ACRAO - ICRAO/N 

AAOl - AAOt/N 

APSP - IPSW/H 



AINSTR - TiNSTfi/N I 

I 

AAOMIN - .TADMIN/N I 

I 

AEQUP - TEQUIP/N I 

I 

AFACIL ■ TFACIL/N | 



100031*1 I OS 

I AIMOMT - TMOMNT/N I 

1 I 

I ASUPPL • TSUFFL/H I 

I I 

I AMISC - TNISC/N I 



I 

I 

THIS ROUTINE 
CALCULATES EQUIPMENT 
PURCHASE ANO 
0EPRECIAT10N 
SCHEDULES FROM USER 
SELECTEO POLICY 
OPT IONS 

I 
I 
I 

- PURCHASE POLICIES 
ANALYSIS 



IF OUTSlOE THE RANGE 



->• 



POLICY I - PURCHASE 
TOTAL AVERAGE RQMTS 
(N FIRST YEAR OF 
EQUIP LIFE 

I 

lO.O**— >1 

)30 I '^OTE 01 

• ••••••••«• 

• CONTINUE • 

1 

I - 

(ooo^oo) I NOTE on 

• BEGIN 00 LOOP • 

• 93A I - 1, N, • 

• LOFEQ • 



I 



10 



I TEMPI • 0.0 I 

I I 

I INOEXl - LOFEQ * I 

I I - I I 

• • 

I 
I 

lOOO^OSl I NOTE 11 



• AEGIN 00 LOOP • 

• %32 J - I. INOEXl • 



->l 
I 



I TEMPI ■ I 

I REQUIPtJ) » TEMPI I 



ENO OF 00 
LOOP? 



1 0003^^1 1 


0* 1 


(000^0*1 1 I 








i COHPUTEO 


CO TO 1 1 


1 PEQUIPd ) - 


1 FOR PURCHF 1 1 


1 TEMPt/LOFEQ 








1 930 


10. 0« 1 1 


1 


1 iKO 


10. U 1 1 


1 


1 550 


11.01 t 1 


1 


1 5*0 


11. M 1 1 


53* ♦ 15 


1 570 


12.12 1 1 


• • 


1 500 


12.13 ( .1 


• • 


1 590 


12. lA ( 1 


• • 


1 *00 


12.15 1 1 


• EMO OF 00 


1 *iO 


12.1* 1 1 


• LOOP? • 



POLICY 2 - PURCHASE 
TOTAL PERCENT OF 
MAXIMUM 

RQMTS IN FIRST YEAR 
OF EQUIP LIFE 

10.0* >• a 

5A0 I NOTE 1* 



(000A12 ) I NOTE 17 

• BEGIN 00 LOOP • 

• 5A« I -. I, N, • 

• LOFEQ • 



I IHOEXl •• LOFEQ * 
I - I - 1 



•EGIN 00 LOOP 



• 5*2 J ■ 

• • • • • 



INOEXl « 
• • • • • 



FALSE • • 

• T£MP1 .LT, 

• REQUlPIJt • 



(OOOAl*) 

• 

I TEMPI - 



EQUIP(J) I 



ENO OF 00 
LOOP? 



I PEQUIPd ) - I 
I TEMPMPCTPCH I 



END OP 00 
LOOP? 



.12. i7. 
... *20 



.12. 17. 
... *20 



ERIC 



IV-41 



TAEG REPORT NO. 12-3 



04/11/79 
CHART T!TLI - MOCIOUKES 



AUTOflOM CHART SET 



POLICY.) - PunCHASf 
UP TO AVENAOE NOMTS 
lUT 

NOT exCflOIMC NOMTS 
IN YR OF PURCH 

10.0* >• 

J50 I NOTE 0 



I 

*l 

(000424) I NOTE 0 

• •••••••••i 

• RECIN 00 LOOP 

• 991 I - I, N« 

• LOPIO 



I 

II. II >| 



I INOEXl 
i I 



TEKPl - 0.0 ' 
LOFEO ♦ 



I 
I 

C00b427) I NOTE 0 

• ••••••••• 

• tECIN 00 LOOP 

• 9S2 J - I, INOEXl 
«••••••••• 

I 

I Tii*pi - I 

I RIQUIP(J) ♦ TfPlft I 



ENO Of 00 

L0OP7 



l-YES 
I 



10004)0) I 



I PEOUIPII ) M I 
1 TinPl/LOFKO I 



coD04)i) • oa 



• PEQUIPUI • TRUE 

.Lf. •-♦ 

• REQUIPdt • I 



(00O4I2) I 



11 
II 



0^ 



I TflNPl - PfOUIPd) I 

i I 

I INOIXI - LOPIQ « I 

1 i » 



I 



— >• 

(000434) I NOTE 10 

• •••••««««• 

' • BECIN 00 LOOP • 

• »SA J - 1 « INOEXl • 

I 

11.17 >| 

I 

• U 
• • 

TRUE • • 
• TENPl .LE. 0. • 



(00043A) • 12 



• • FALSE 
• TENPl .CT. • 

• aEQUIP(J) • 



I 
I 

(0004)*) 1 I) 

I PEQUIP(J) - t 
I REQUIP(J) t 



I 



• • FALSE 

• TENPl .Lr. • ♦ 

• REQUIP(J| • 



; PEOOIPIJ) - TEMPI I 



r I* 



I TENPl - TENPl 
t P.£OUIP(J) 



• NO 

ENO OF 00 

L00P7 • t * 

• • 1 1 

• • I 



trES 



II . 
II . 



-ll.OI*>->« 



• • NO 

• ENO OF 00 •-• 

• L00P7 • I 



POLICY 4 r PURCHASE 
UP TO PCT OF NAX 
RQNTS lUT 

NOT EXCE'EOiNC RQNTS 
IN YR OF PURCH 

10.04— •>• 

)A0 I NOTE 19 



I 

- I 

(00044)1 I NOTE 20 

• •••••••••• 

• lECIN 00 LOOP • 

• )At 1 - It N( • 

• * LOFEQ • 

• •••••••••• 



I 

12. II >| 



I TENPl - 0.0 I 
I I 
1 INOEXl - LOFEQ * | 
I I - I I 
• — • 

I 
I 

(000441) I NOTE 22 

• •••••••••• 

• lECIN 00 LOOP • 

• )A2 J - I, INOEXl • 

• •••••••••• 



I 



FALSE • • 

• TENPl .LT. 

• REQUIP(J) • 



I TRUE 



(00044f) I 24 
• " « 

I TENPl - REOUIP(J) I 



ENO OF 00 • 

L00P7 • 



t PEOUIPII) - I 
I TENPl«PCTPCM I 



I / 
/U.Ol 



id 

ERIC 



IV-42 



c30 







c 
















TAEG REPORT 


NO. 12-3 


























04/21/79 




AUTOFLOW CHART 


SET - 


FACE 12 








CHART TITte • FROCfOURES 






































/ 570 / 










1 1.2* >• 






1 

10.0* — >t 










1 






1 12 










10004921 • 01 






. . . 


- OEFRECIATION 








• • 








FOLICIES ANALYSIS 








• • 






.10.01. 










• FEOUlUn • TRU€ 








10.19* — >• 














... 930 


*20 1 17 








• REOUlFin • 1 
















• m :> I 








1 CONFUTED CO TO i 








• • 1 








1 FOR dEFF 1 








IFAtSE 1 








1 h30 12.18 1 








1 1 








1 *40 13.01 1 








1 t 






/ 9ao / 


1 hSO 13.M0 1 








1 1 








1 **0 13.14 1 








1 1 






1 


1 *70 13.19 1 








1 1 






10. 0* >| 


1 *10 , 13.1* 1 








{0004931 1 02 1 






1 13 


1 MO U<17 1 








• 1 








1 7tf0 13.11 1 








1 TEMFl • rEOuiriii 1 1 




• 




1 710 13. 1» 1 








1 1 i 






.10.01. 










1 INOEII • COFEO * 1 1 








1 








1 1-1 II 






... 930 


1 
















1 








1 1 








IF OUTSIOE THE RANCE 
1 








1 1 
< {000499) 1 NOTE 03 | 








1 








««••••••••• 1 






— ■■• ' 


1 








• BEGIN 00 lOOP • 1 








FOLlCr 1 - 0EFREC1AT6 








• 96* J » 1* INDEXI • 1 






/ 9fO / 


LIN€ARLY OVER 

















LIFE OF EQUIFMENT 

^ 1 








12,10 >| 1 






1 

10. Oh >| 


12.17«~>| 








1 , 1 






1 14 


*30 • 1 NOTE 11 








• 04 1 






... 










* * 1 








• CONTlNUf • 








* * 1 






.10.01. 










• . . • TRUE i 








1 








• TEHfl .LE. 0. 1 ♦ 






... 930 


1 








• - • II 








(00047*1 1 19 








* * II 














• • I I 








1 TEHF2 » LOFEO 1 








* * II 
















IFALSe 1 1 








1 
1 








1 1 1 

I 1 1 






/ *00 / 


(000477) 1 NOTE 20 








1 1 t 






— 


• ••••••«••« 








1 1 1 






1 


* lEClN 00 LOOF • 








t 1 1 






10.0*— >| 


• 432 1 • li LOFEO * 








1 1 1 






1 19 










{0004971 •09 II 






... 


1 








• • II 








♦ ^ >| 








• • II 






.10,01. 


1 1 21 








FALSE • • II 














* 








... 9 30 


1 1 TEHFl -I 1 








1 • REQUIFIJI *. 1 1 








1 1 1 








1 * * -II 








1 1 FOEFin • 1 








1 * * * i . 1 








1 1 TENF1/TEMF2 1 








1 * II 
















1 {TRUE 1 1 


12,07*— >• 






.1 1 








1 - ' t II 


1 


09 




1 1 








1 1 II 






/ 410 / 


1 1 








} 1 II 


1 TEHFl • TEMFr- 


1 




1 432 * 22 








1 1 It 


1 FEOUlFIJt 


1 


1 


I • • 








1 1 II 






10.0* — > 1 


I • • 








1 {0004971 j. 0* 1 1 


1 




1 1* 


1 NO * * 










>| 






♦ • ENO OF 00 • 








1 1 FEOUIFU) • 1 1 


1 . 




* . 


• L00F7 • 








t 1 KtOUIF(J) 1 1 


9kfc • 10 




. fo.oi. 


• • 










• • 






- • • 








1 I I 


• • 




... 930 


• 








♦ >l 1 


• • 


NP 




lYES 








1 t 


• ENO OF 00 






1 








• 07 1 


* L00F7 * 


j 




1 








* * t 


• • 


1 




I 








* * 1 


• • 


1 




I 








• • FALSE 1 


• 






I 








• TEHFl .LE. •-♦ 1 


1 YES 


12 , 




1 








• REOUlF(J) • 1 1 




04 . 












• • 11 


1 














• • II 


1 
1 






.13.20. 
















... 730 








1 . 0» . 
















1 .... 


9kf • 11 














1 


• • 














1 


• • 














1 


• • 


NO 












(0004911 t 01 


• ENO OF 00 
















• L00F7 • 


■'! \ 












1 FEQUIF(J) • JtnH 1 


• • 






























1 


• 








CI 






1 


IVES 














1 




21 . 












! / 
















/12.09 


] 
















,! 12. 17. 
















... *20 


















IV-43 










FRK 

































CMMT Tiai - mCUMMlS 



TAE6 REPORT NO. 12-3 



MCE 19 



*90 



POLICV 1 . OlfffiCIHTI 
fOUIPMtMT 



12.17- 
*4« 



I 



Nori 
ccmriNUf 



I 
I 



I 



lUfI - 0.0 



I 
I 

I0004t«l I NOTE OJ 
••••••••••• 

• lEfilHI 00 LOOP • 

* *«2 i - 1, LOPEQ • 



I 



EM} Of DO 
LOOP? 



(YCI 

I 
I 
I 



1000449) 1 



I POEPtU " t 

t ILOPE0)/l(/ff I 

I 
\ 

IOO«4fO» I NOTE 07 

• teCIN DO LOOP • 

• I • 1, LOfEO • 



I 



I POEP(I) • I 

I (LOPio - n/i(M ♦ i 

I POEPd > 1) I 



eNO*^ 0 

L00P7 



POLICY J - JET 
DiPMECIATION ICKfOULE 
TO PULLY OCPUBCIATEO 
ALL YEAKS 

12.17 >• 

*tO I NOTE 10 

• CONTINUE • 

I , 
I 

IOOO<1T) I NOTE 11 

• lEGIN 00 LOOP • 
« ftS2 1*1, LOPEO • 



I IT 
.12.lt. 



,700 



I 

12.17 >| 



I 
I 



POEPdi -1.0 1 

I. " • 



I II 

.12. ll! 



TIO 



ENO OP. 00 
LOOPT 



I 

12.17 ^>| 



12.17 >l 

I 1« 



I M 



12. IT >| 

I It 



.12.it. 
... 410 



I 1« 

!i2.ii! 



- COST PACTOHS 



12.22*-->* 

TIO 1 



20 
— • 

I 



(000912) I NOTE 22 



lECIN 00 LOOP 
T*0 I - L. 20 



CKAOd) 



I COPMT(L) - 0.0 



(0009U) • 29 



ENO OP 00 
L00P7 



I NOTE 2ft 



(0009U) 1 
I TOTAEO - 



2T 
— • 

I 



(000917) I NOTE 21 

• BEIlN 00 LOOP • 

• TfO I « 1, N • 

I 

19.0T >| 

t 29 

• • 

I . CSTUO(l) - I 

I ((ITUONW(I) * I 

1 SNlMIIC(I)|/92) I 

I •STUOIL i 



I CTRAVdl « 

I ITUOd )«(|TCITI 

I STCST2*(1.0 - 

I .««ARATE»I 

I 



IV-44 



TAEG REPORT NO. 12-3 



0%/23/7i AUTOFLOW CH*«T SET - 

CHART TITLf - PROCEOURES 



1000)20) I 



I CINSTRU) • 
I RINSTRMI*INSTSL 



CAOMINM) - 
RADM|MlII**OMSL 



I TPEOUP - TPEOUr 
) PeOUMU) 



IN0EX4 



IN0EX4 



I rEOUir ( I )*EOUNTC 



I LIFERV - I ♦ I 
1 LOFEO - N - I I 



I00052T) • 04 



TRUE • • 
• LIFERV .Lfc. 0 • 



I 
I 
1 

<000)2t) I 0) 
• • 

I Rveo • Rveo * I 

r CEOUiril) I 
I •POEP 1LIFERV» I 
• — • — • 

I 

— > I 

740 i NOTE 04 

• •••••••••• 

• CONTINUE • 

• «••••••••• 

I 
I 

lOOOMO) I 07 

• • 

I TOTAEO • TOTAEO 
I PEOUirin 



AEOUir I I ) 
TOTAEO 



I tAEOUi» - TAEOUr 
I AEOUl)»(II 



I LOOSLT - I 
I LOFEO 



)000^1*« • 09 



TRUE • • 
• LOiSLT .LE. 0 • 



(000)Y<() 



10 



I TOTAEO ■ TOTAEO - I 

I r€OU|P(LOBSLTI | 

I I 

I AEtiUIPd ) - I 

r TOTAEO I 



r.Eyui^MJ ■ 
CEguiPtit • 

AEOUIPI I ) 

l.FAClLtll -■ 
RFACim »«CPSOFT 

tlNONTUI - 
IMONNTI r)*ClNO 



TRUE • 
• I .CT. 



I CTNOHTUt • I 

I CIHnHTtI ) • CUIHO I 



->l 
I 



csuppun - 

RSUPPL \ I »»CSUPP 



cwisc( n - 

RNISCI n*CHIS 



I 
I 

- COST SUNHARHATION 

I 
I 

1000)47) I I 



OISCNTI II-' 
12.0 » 
ORATE) 
/(2.0*l 1.0 » 
ORATEI««l* 

INFLAT I I t - 
I 2.a«( l.O • 
INRATE I } 
/\2.0 * iNRATf I 

" 1 
I 



NSTUO • NSTUO ♦ 
CSTUOl I ) 

OSTUO • OSTUO ♦ 
CSTUOl I ) 
•OISCNT I I ) 

ISTUO - ISTUO * 
CSTUOl I ) 
• INFLAT I I ) 



FSTUO - FSTUO 
CSTUOl I t 
• INFLATt t ) 

•DiscMTi n «■ 

NTRAV - NTRAV 

CTXAVI I ) 

OTRAV • OTRAV 
CTRAVIl ) 
•OlSCNTi I ) 



ITRAV - I TRAV • 
CTRAVt I ) 
«INFLAT( I I 

FTRAV - FfRAV • 
CTRAVt I t 
•INFLATt I ) 
•OISCNTtU 

NINSTR - NlNSTR 
CINSTRI I ) 



OINSTR - OINSTR 
CINSTRt I ) 
«0)SCNTt I ) 

IINSTR - IINSTR 
CINSTRI ) I 
•IMFLATI I ) 



(000)401 L 



1000)62) I 



(000)44 1 I 



FINSTR - F1NSTR 
CINSTRIII 
• INFLATd I 
. •OtSCNTI I ) 

NAONIN * NAOHIN 
CAOMINI I ) 



OAONIN - OAOMtN 
CAONINd I 
•OISCNTI I I 

lAONtN - lAONlN 
CAOMmi I ) 
•INFLAT ( I ) 



FAO><IN • FAOMIN 
CAOHINI I ) 
• INFLAT( I ) 
'OiSCr(r( I ) 

NFACIL - NFACIL 
CFACIL ( I ) 



OFACIL - 0F4CIL 
CFACIL ( I) 
• OISCNT I I ) 

IFACIL > IFACIL 
CFACIL I I I 
•INFlAT( I ) 



1000)41) I 



FFACIL - FFACIL 
CFACILI I ) 
• INFLATd ) 
•OISCNTI I ) 

NSUPPL • NSUPPL^ 
CSUPPLd) 



OSUPPL • OSUP^L 
CSUPPLd ) 
•OISCNTI I ) 

ISUPPL • ISUPPL 
CSUPPLd ) 
•INFLATI I ) 



(000)72) I 



FSUPPL ■ FSUPPL 
CSUPPI I I ) 
•INFLATI I ) 
•OISCNT I I I 

NMISC - NKISC • 
CHISCI I I 

OHISC - DMISC • 
CHISC I I ) 
• DISCNTd ) 



IHiSC - IMISC 
CHlSCd ) 
•INFLAT I I ) 

FHISC - FHISC 
CHISCI I ) 
•INFLATI I ) 
♦DISCNTd 1 



I 



1000)77) • 



1000)71) I 



I) 
01 



21 



I CfOUlPd ) • I 

I CEOUiril) - RVEO I 

I \ 

\ CIWOHTtI) - I 

I CtMOMTd I - RVIN I 



I 



ERIC 



IV-45 



TAEG REPORT NO. 12-3 



04/»/Tf ^ AUTOflM CHAKT IfT • 

CMMT TlTll • MOCQDU«IS 



1 NIQUir - NfCUl^ 
I CEQUIPUI 

I CfOtllPdt 
I •OlSCMTtlt 



1000St2) I 



t •INVUTMI 

I CEQUI^lII 
t •INfLATIII 
I ^ISCNTtn 

I NINO - NINO ♦ 
I ClNONTtn 



(000919) I 

I OINO • OlM) ♦ 

I CINONTtll 

t •OlSCNTtll 

I 

I UNO - UNO ♦ 

I CINONTin 

I ^ «INfLATI|) 

I flNO - FINO * 

I CINONTIII 

1 •OlSCNT(l) 

I •INfLATtI) 



(0009tt) I 



COPTOTII) 
CSfUOMl ♦ 
CTKaVMI ♦ 
CINSTR(I) 
CAONlNllI 
CfACILtl) 
CINOMTI t i 

CNISCIII ♦ 

ccouirin 



I0009«0) I 



09 



i C6«A0<I} - t 

I cofTOTi ii/MAOl n I 

I I 

I CST!N(I ) - I 

I COf TOTI n/sTuoin I 



I 



I CSfll) - 

I corroTt n/ftf ( I 



(0p09«2) • 07 



0* 
• 

I 

I I 



CWO OF DO 
LOOPT 



Ivss 



NO 
I 



11 
29 



(000991) 
APCQUP - TMtUf/N 

I 

I AAIQUP - TAiOU^/N 

I 

I NFACIL « NfACa ♦ 
FACOST . 



0*1 



OFACIL - OFACIL ♦ 
t FACOST. 



I IFACIL - IFACIL «- 
I FACOST 



I FFACIL - FfACa ♦ 
I FACOIT 



I 
I 

I000999> I 

I SNAOC - NSTUO ♦ 
I NTRAV ♦ NINSTR ♦ 
I NAONIN ♦ NFACU ♦ 
I NSUffL ♦ NNISC ♦ 
I NEOUIf ♦ NINO 

I 
I 

(OCO&OO) I 



I SOAOC - OSTUO ♦ 

I OTRAV ♦ OINSTH ♦ 

I OAOHIN ♦ OFACIL ♦ 

I OSUffL ♦ OMISC ♦ 
I OEOUIf ♦ OINO 



I 
I 

(oooAon 1 ' 



1 SIAOC - ISTUO ♦ 

I ITRAV ♦ IINSTR * 

I lAONIN ♦ IFACIL ♦ 

t ISU^PL ♦ INtSC ♦ 
I lEOUlP ♦ UNO 



(000A02I I 



SFA.QC - FSTUO ♦ 
FTRAV ♦ FiNSTR ♦ 
FAONIN ♦ IFACIL ♦ 
FSUI*PL ♦ FNISC ♦ 
fCQUir ♦FINO 

CNAOC - 
SNAOC/ IMAO 



IOOOAO<V) I 



GOAOC « 
SOAOC/IGAAO 

61A0C - 
SIAOC/IMAO 

CFAOC - 
SFAOC/IGAAO 

UNAOC - 
SNAOC/lAASf N«N) 

I 
I 



I U0AM - 

I SOAOC/ 1AASIN*N) 

t ' u'lAM - 

I SIAOC/iAASlN*!*) 

I U^AOC - 

I SFAM/IAAS|N«N) 

I fNAOC - 

) SNAO<;/|^Sf 



19 

I 



1000AI2I 



— • 

I 



I fOAOC - 

I SOAOC/lfSf i 

I I 

I riAOC - I 

t SIAOC/lfSf I 



PFAOC - 
SFAOC/IfSf 



ORATE <* 
ORATE«IOO. 



INRATi - 
lNIIATS«tOO. 



I 



WRITE INPUT TfC»«<ICAL 
FACTORS *^ 



900 • It 



• • TRUE 
• .NOT. RPTl • 



rFALSE 



1000421) I 



19 



/ WRITE TO 0£V / 
/ 4 / 

/ VIA FORNAT / 
/ 1000 / 

/ FRON JHt LIST / 

I 

1000421) I NOTE 20 

• ••••••••• 

• LIST - COURSE* 

• ORATE. N. INRATEt 

• ICKAOlDtl - 

• l.lO)t 

• IMAOtll.I - 

• U.20). ARATI. 

• TSfOSO. TLiNGHt 

• fiiSP. RCRATEt 

• ^EOTSfO 



I 
I 

(000419) ( 21 

/ WHITI TO DfV / 

/ 4 / ' 

/ VIA FORNAT / ' 
/ 1010 / 

/ FRON THE LIST / 



>• 

(000419) I NOTE 22 

• ••••••••«• 

• L'IST - ARCVTNt • 

• LdFEQt WSCHOP. • 

• INOOEVi iNTSPOt • 

• PUINOf ANTSFOi • 

• UPOATEf SOFTSTt • 
• • EVIM. SQFTINi • 

• SUfSFO, SQFTAN, • 

• SUfSTO. ^CHFi • 

• NSCSfO. OffFi • 

• NSCSTO • 



WRITE INPUT COST 
FACTORS 



*000449) I 2h 

I WRITE TO OEV / 
/ 4 / 

/ VIA FORNAT / 

/ toot / 

/ FF.ON THE LIST / 



10004*9) r NOTI 29 

• LIST - COURSEi • 

• ORATEi Ni INRATEi • 

• (COPNTlD.l - • 

• tttOli • 

• ICOPNTID.I - • • 

• ltt2Dli STUOSLi • 

• CINOi STCSTli • 

• CSUPP. STCST2i • 

• CNIS, INSTSLt • 

• PCTPCH. AONSLi • 

• CPSOFTi EOUNTC . • 



I 

W«ITE SUNNARV OF 
ANNUAL TECHNICAL 
RESUifS 

I 

|< 

r 

920 • 24 



• • TRUE 
• .NdT. RPTl 

• • . I 



IFALSE . 14 
I . 02 



(000492) I 



27 



/ WRITI TO OEV / 
/ <, * / 
/ VIA POftftAT / 

/ tool / 

/ FROM THE LIST / 



JV-46 



04/23/ 15 



TAEG REPORT NO. 12-3 

AUTOPLOM CHART SET * 



PAGE 14 



CHART TITLE - PROCEDURES 



\i,21 >• 

1000492) I NOTE 01 

• ••!»••••« 

LIST - COURSE, 
ORATE, N, INRATE, 
ACRAO, AASIN, 
AAOD, APSP, 
AINSTRi AAOMIN, 
AEOUP, APEQUP, 
AAEQUPr AFACIL* 
AIMOHT, ASUPPL, 
AM ISC 



I 



MRITE ANNUAL 
TECHNICAL RESULTS 



(OOOTIT) I NOTE 0 

• •••••••• 

LIST - COURSE, 
ORATE, N, INRATE, 
lOIGITIli.I - 

1.91 , 
(CRAO(I).I - 

1.91, 
tSTUOilltl - 
1. 91, lAOIin.l.- 
1,91, tFSPin.I - 
1.5). 
tRINSTRU).! - 

1.9) . 
tRAOMINd).! - 

1 .9). 
tREQUirin.l - 

1.91. 
.( f EQUIP ( D.I • ° 

• •••••••• 

I 
I 

10007171 I NOTE 09 



1.9). 
lAEQUIPd),! - 
1.9). 

irfacilcd.i • 
1* 91. i immnti i ). 

I - I. 
9).tRSUPPL(I ). 
t - 1, 
91,tRH|SC(ni I - 
1, 9) 



I 



1000724) I 07 
/ WRI TE TO OEV / 



11.19). 
(RAOMIN) I > . ) - 

11.19). 
(REOUIPf 1 )i I - 



1 






(000739) 1 NOTE 12 




(00Q7**I 1 NOTE 


17 


729) * 04 




• 


• ••••••••« 








• • 




• 


LIST • COURSE. • 


1 • 


14.20). 


• 






• 


ORATE. N. INRATEi • 


1 • 


(PEOUIPI 1) . I - 


• 


• 


TRUE 1 


• 


(OICIT 1 I ) . 1 - • 


1 • 


14,20). 




N .LE. 9 


1 


• 


11.19). • 




(AEQUlf (|).l - 1 


• 


• 


1 1 


• 


iGRAOin.I - • 


1 • 


4. 


• 


• • 


1 1 


• 


11.19). • 


1 • 


20).(RFACIL(I), 


• 


• • 


1 1 




ISTUOt I ), 1 • • 


1 • 


I - 14. 




• 






t 1. 19) , • 


1 • 


20) . ( INOMNT( I ) , 




IFALSE . 


14 . 1 


• 


(AOIl I) > 1 - • 


1 • 


I - 14. 




1 


11 . 1 


• 


11.19), • 


1 • 


20) . lRSUPfL( 1 ) . 


• 


1 




• 


IfSPUI.I - • 


1 • 


I - 14. 


• 


1 


940 1 


• 


11.19). • 


1 • 


20) . IRniSCf I ) . 


• 


1 




• 


IRINSTRt 1 ). 1 - • 


1 • 


I - 14. 201 


• 



930 • 02 i 




/ VIA FORMAT / 




1 


* * 1 




/ 1009 / 




1 


* * 1 


1 


FROM THE LIST / 




(000739 ) 1 NOTE 


• • TRUE 1 










• '.NOT. RPT^ 1 




I 




* 11.19). 


. • ' • 1 1 




1 




* (FEgUIP( D.I - 


• • 1 1 




1000724) 1 NOTE 


01 t 


* 11.19). 


• • II 


• 


• •••••••• 


* 1 * 


• (AEQUlPd)*! - 1 


• • ■ • • 1 


• 


LIST - COURSE. 


• 1 


♦ li 


{FALSE . 14 . 1 


« 


ORATE. N, INRATE. 




• 191 . inFACIL( D 1 


1 . It , 1 


• 


(OIGIT( I ) . 1 - 


• 1 


; 1 - 11. 


1 .... 1 


• 


4.10). 


• 1 


• 19) .( IMOMNTd ). 


1 940 1 


9 


(GRAOd ),| - 


* 1 ' 


• 1 - u. 


1 1 


• 


4.10) . 


• 1 


• 19) . (R^UPPL( I ) . 


1 1 


• 


ISTUOID.I - 


• 1 


• 1 - II. 


(000717) 1 03 1 


• 


4,10), 


• 1 


• 19).(RMISC(I). 




• 


(AOB( D.I - 


* 1 


• I - 11. 19) 


/ WRITE TO OEV / | 


• 


4,10). 






/ 4 / I 


• 


(PSP( I ) . I - 


• 1 


1 


/ VIA FORMAT / | . 


• 


4.10). 


• 1 


1 , 


/ 1009 / 1 




{RINSTR( Di 1 - 




\ 


/ FROM THE LIST / | 




4.10). 


• 1 


(000743) • M 




• 


(RAOM|N{|).I - 


• 1 


• • 


1 1 


• 


4.10). 


• [ 


• • 


' 1 1 


• 


{REQUIP( D.I - 


* ' 1 


• • 



(010724) 1 NOTE 09 



10007 34) • 10 



(FALSE . 14 . 
I . II . 



(000739) \ \\ 

I WRITE rC OEV / 
/ '4 . / 

/ VIA FORMAT / 
/ 1009 / 

/ FROM THE LIST / 



N .LE. 19 • ♦ 



4. 10) . 


* i 


IFALSE 




(PEQUIPI D . I - 


* 1 * 


\ 


t< * 


4.10). 


* i 


\ 


1 • 


(AEQUIP(D.I - 


• 1 


\ 


j • 


4,10), IRF 


« 1 




1 « 


ACIL( I ), 1 - 4, 


• 1 


\ 


1 • 


10), dMOMMTtl) , 


• 1 

• i 


1000744) 1 19 


1 • 


1-4, 

IO),(RSUPPLd), 


• 1 


/ WRITE TO OEV / 


1 • 

1 * 


I - 4, 




/ ^4 / 


1 • 


10) , IRHISCI 1 ) , 


• 1 


/ VIA FORMAT / 


1 • 


1 - 4, 10) 


• 1 ^ 


/ 1009 / 


1 * 



/ FROM THE LIST / 

I 
I 

(000744 1 I NOTE 1 

llST - COURSE, 
ORATE, N, INRATE, 
(OIGI Tl D ,1 - 
14,20) 



(REQUIF(D,I - 
• •••• •••• 



(GRAOd ) , 1 - • 


i * 


PL, 


DSUPPl, 


« 


14,20), • 


1 • 


ISUfPL 


, FSUfPL, 


« 


( STUOd ) t I - • 


1 • 


NMISC 


, OMISCi 




14,20), • 


1 • 


IMISC 


, FHISC, 




lAOId } , 1 - • 




SNAOC 


, SOAOC, 


« 


14,20), • 


1 • 


SIJ^OC 


. SFAOC, 


• 


(PSF( ( ) , I - • 


1 • 


GNAOC 


, COAOC. 


« 


14 ,20) , • 


1 • 


GIAOC 


r GFAOC. 


« 


(R^NSTRd),) - • 


1 • 


UNAOC 


UOAOC. 


« 


, 14,20), • 




UIAOC 


UFAOCt 


« 


(XAOMIN( I ) , I - • 


1 • 


fNAOC 


FOAOC. 


« 


14,20)i • 


1 ' * 


PIAOC 


FFAOC 


« 



WRDE SUMMARY OF 
ANNUAL COST RESULTS 



• • TRUE 

• .NOT. RPT9 



/ WRITE TO OEV 
/ 4 
/ VIA FORMAT / 
/ 1002 / 

/ FROM THE LIST / 



(000774) I NOTE 20 



ORATE. N, INRATE, 

NSTUO. OSTUO, 

ISTUO, FSTUO, 

NTRAV, OTRAV, 

ITR4V, FTRAV, 

NINSTR, OINSTR, 

IINSTR, FINSTR, 

NAOMIN, OAOMIN, 



• fEqUlf, FEQUIf, * 

• NFACIL* OFACK, • 

• IFaCIL, FFACIL, • 

• NIMO, OIMO, IIMO. • 

• FIMO, NSUf • 



(000774) 1 NOTE 2t 
• ••••••««« 



! ' 

WRDE ANNUAL COST 
WSULTS 

I 1 
I 

I 

I / 



IV-47 



J. TAER REPORT NO. 12-3 

04/II/T5 AUTVLOW CHAItT SET - ^ MCE IT 

' CHAKT TITLI - PHOCtOMES 



1. 1 

I 



TRU( 

I 



9*0 



(OOOIOII I 02 

/ MRITI .TO DBV / 
/ » / 

/ VI A rOAHAT / 
/ 1004 / 

/ THS LIST / 



) 

(OOOIOII I NOTE 0 
• • • • • • • 

LIST • COURtEt 
MATft Nt IMRATEt 
(OICITI Itt I • 

i.SIt 
(CSTUOCIItl " 

i>9). 
(CTRAyf!),! - 

(ClNSTRIDil - 

i>S>. 
(CAONtNtnil • ' 

1.9). 
(CIQUirUltl • 

i 1 9 J t 
(CFACILIUd - 

LCtHOMTUItl • 



r 1 • 

t 

(OOOIOI) t f(OTf 0' 



itS). 

(ctuppLinti • 

i.l). 

(CMisctn.t • 

1*9}. tCO^TO 
TUI. I n i> 
9).ICMAD(I)t \ 

i» s)t(csn»in. 
I • I. 

S)»(CSP(I)t I - 
i. S) 



I 



(OOOtl*) • 09 



rauE 

I 
I 
I 

. IT . 



(OOOtlT) i 0* 
/ WftlTI TO OIV / 

/ » / 

/ VIA fOII«IAT / 
/ iO«4 
/ HIO« TMf LI IT / 



(000117) I NOTE I 

LIST • COlM^e. 
DRATE. Nf (MATE. 
(0(6IT( I ).I 

*»10N 
(CSTUO( I ) . I 

A.iO), 
(CTftAV( I ), I 
*>10|, 
tC{NSTR(I)iI 

*>I0). 
(CAOMIN( II. I 

*f 10) > 
tCEQUI^(I)tl 

A.lOtt 
tpf AC I L ( I ) . I 

*.10), 
(CIN0NT( I )t 1 
• •••••• 

I 
I 

(000117) I NOTE 

*.I0), 
tCSUPPLdl.l 
*>10)> 
(CNISC(l)iI - *il 
0).tCO^TOT(I)> 

I - *t 
10) . (CCRADd ). 
I - 

10). tCSTIN( 1 ). 
I • 

10)>(CSP(I). I - 
*. 10), 



• • • • 



I 



(00012 9) • 



TRUE 



(00012*) I NOTE 12 

• tt,19)t • 

• (CiUfPL(l)tl < • 

• 11.19)t (CNISClW • 

• ).I - A • 

• 11.19)((C0fT0T(T) • 

• ,1 - H • 

• 11.19)t(CIMA0tII*. • 

• I ■ }\ • 

• it.i9).(csriN(ri', • 

• I - ' |,J • 

• ii.i9)f (cspdri • 

• I - 11(19) ' 



'J. 



IS 



/ MITE TO DCV / 
/ * / 

/ VIA FORMAT / 
/ 1004 / 

/ fRON THE LIIT / 



(0O0IS9) I — »«Tt 



• • 


1 1 


• LIST > COURSE. 


• 


• • 


1 1 


• ORATEt N> INRATEi 


• 


• • 


1 1 


• (OlCIT(l)il - 


• 






• 1**20)( 


• 


IfALSf . 


IT . 1 


• tCSTUOin.l - 


• 


1 


II . 1 


• (*t20)* 


• 


1 




• (CTRAV(1)>I • 


• 


1 


9*0 t 


• l*t20). 


• 


1 




• (CINSTRfH. I - 


• 


t 




• , 1*.20) . 


• 


(00012*) 1 


10 1 


• (CAONINtn.I • 


• 






• 1**20), 


• 


/ WITE TO MV 


/ 1 


• (CSOUIfdId ^ 


• 


/ * 


/ i 


• l*tZOIt 


• 


/ VIA fORNAT / 




« (CPACIL(1)>I • 


• 


/ 1004 / 




• l*)20)t 


• 


fROM THE LIST / 




• (C;NONT(lltI - 


• 


1 
1 




1 ' 




(00012*1 1 NOT! 


11 t 


(000IS9) 1 NOTE 


t* 


• ••••••••• 








• LIST • COURSCi 




• 1*>20}* 


• 


• ORATE. N. IN«ATI( 




• (ClUffL(I)tl - 


• 


• (DI6|T( Dtl - 




• l*t2b). (CNIICfl 


• 


• lltl9)t 




• )>l - 


• 


• (CSTUOIDtl - 




• l*t20)>(C0fT0Tin 


• 


• .11.19). 




• >I « 


• 


• (CTRAV( I III* 




• 1*.20),(CMA0(M. 


• 


• 11.19). 




• 1 - 


• 


• HC|NSTR(l)tI • 




• 1*.20| . (CIT1H( I ) . 




• 11.19)> 




• I •< 


• 


• ICAOniNtDtl - 




• l**20)«(Cir(llt 




• 11.19). 




• I - l*t20} 


• 


• ICEQUIflD.I • 








• 11.19). 




I 




• (CPAClL( Dtl • 








• 11.19)t 




1 




• tCINOnTID.l « 




MRITE H1IIIM6 




• ••••••••• 




' ITATIMMT NUMIRI 





9*0 

TRUE • 
• .N 



(0001*7) • 



TRUE • 
< • 



<LE. 



/ WRITE TO DEV 



20 
/ 



(0001*9) i 



21 

I 



(OOOITO) I -22 

/ MRITE TO DEV / 
/ *•' / 

/ VIA f>ORMAT / 
/ 9001 / 

/ fRON THE LIST / 



I 
I 

(000170) I NOTE 23 

0 LIST - • 

• ((pATA(|),MATI(l) • 

• )tl - l.K) • 

• ••••• 

1T.1I«— >} 

990 • 



24 



«NOT. £aIT 



tPALie 

I 

I * 

I 
1 
I 

'•(000179) I 29 

• MIT • 



2 

01 



IV-48 



TAEG REPORT NO. 12-3 



CHART TITLE - SUBROUriNE HI SSNC I NUMI « ICODE ) 



AUTOPLOM CHART SET - 



03.20*— >! • 



FALSE • • 
• ICODE ,EQ. 0 



i TKUE 

I 

I 

I 
I 
I 

(000I79> I 02 ^ 
.--.Jj. . 

I WNDGO - i I 
• 

I 

_ > I 

/ ' °' 

I HAFaIx) - NUKI I 

I I 

i HAra(K) - ICODE I 

I I 

I K - H ♦ 1 I 

• — — • 

I 
I 

lODOlOB) I 0^ 
• EXIT • 



ERIC 



IV-49 



TAEG REPORT NO. 12-3 



LU LU 
U O 

UJ lU 
Q. Q. 

o o 

0 • 

O >Q 



O 



0* 





' i t 
1 1 


















o 






11 J ^ 


Art 














^ Q. 
























ft— n 


w 
























<JU 






O -J 








I/) UL 




















O 














^ O 




*^ 








l/j 








>• 




















o 


o 






o 








o 




















I/) 








o 








o 




O 


o 






O 








o 












ac 
















O ' 








U 




o 


o 






o 








m o 






























z 


to 






LUCO 


CC O 


o 




to 


o o 








eg o 






i t 










< ^ 






o 


LL 






o 








QC 


O 


o 




LU tsC 


< O 






CD LU 


O O 






ac Q. 


»-4 o 








Z f-4 






z o 


X 






z 








LU ^ 


LU > 1 


> 1 




(/) z 


C/> 


to 




oc z 




O O 




3 < 


»- Z '«t 


z < 


3 O -1 


3 ^ OC 


-1 QC 


a. 




Q. Q. O 


Q. O 


z 




Z 



'-^/in o o o O' 

oom oooj^oooo 

• •^0 •tttttfl* 

O O IT ^ 



I I I I I 



— . 00 
I- z 
o o 

Q. i-i 

— I- o 



z to 

3B Q 
O Q. 
O 

O 
to 3 
O >- 
Q. to 

Q.< 

to,x 

H Q- 



QC Z 
LU 
to X 

o a. 
a. M 

a or 

LL 
^ O 
OL 

M IJJ 

3 LL 

cr 

LU -J 



t 

■CL > 
X LU 
O 

LU 

to O 
QC Z 

O QC 

to a 

QC LU 
X QC 

> UJ 

LU to 
O QC 

r) 
o 
o 



11)11 



cL UJ z 
LU 3 O 
Z •J 

•-^ < 

to 

zoo 

Z Q- 

X QC »- »- 

z z 

• • UJ UJ 
Q Q 
< < 3 3 
X X >- >- 

to to 

QC QC X. ^ 

to to 

to to UJ LU 
Z Z *^ »M 

M _J _J 

Q. a 

K OL Q. 

O O 3 3 
flL OCr (/) (/) 



z 

O 



to 

O 

a 



z z 

LU LU 
O O 

to to 

o o 
to to 



O O 

CM o in f-H o o o 

00»«*OvOOOOOO 

i rg o in ro'rg ^ 

1 in 



i I I I I 



t i t M I I 



to 

3K 



a w »~ LU 
O X 
X Q. i-N 

UJf 



QC 
< 



Z UJ 
LU K 
U < 

QC 

O 

Z UJ 
Z O 

< o 

QC LU 
QC 



LU 



UJ Q 

< H 
QC < 
LU QC 
Q. 

O to 
O 

^ a. 

o 

O Q 
X 3 
O >- 
to to 

to QC 
^ >- 
LU to 
UJ Z 

3B H4 



Q • 

>H to 

>- a 

oc 

to Z 
Q LU 
CL O 

Q >^ 

to 

to 

X LU 
Z UJ 
»-4 U. 



> o 

O Q. 



Z t-t 



LU LU 
LU LU 
LL U. 



Q a car or 
< vo to to 



-I Z 

o o 
a. ^ 

LU < 

to I— < 

< U 

X UJ 

U QC 

OC Q. 

'3 UJ 

a o 



/ ■ . 

TAEG REPORT NO. '12-3 



\ 



z z 

LU LU 
U O 

LU LU 
O. Q. 

O O 
» t 

0 >o 

.fH ■ 

1 I 

LU 
LU K 
K < 

< oc 
z 

HO 

z 

O < 
u ^ 
(/) u. 

z 
o ^ 



o 



o 



o 



o 
00 o 
rg 



O 
o 



00 



f 

o 



t 

o 



00 














rg , 




00 










>- 




















< 






t 




z 






o 


o 


< 




in 


o 


in 








rg 














00 










o 










a 






t 


t 








o 


o 


LU 






o 




CO 






rg ' 




oc 










3 










o 










a 






t 


f 








o 


o 






m 


o 


rn 








rg 
















Z 










LU 00 




t 






00 




o 








rg o 


rg 




1 1 


tff . 


rg 
















a 


oc 








o 


o ' 








oc . 




t 


t 




LU OC 


o 


o 


o 




CO LU 




o 






X Q. 




rg 






3 










Z CD ' 










Z 


00 oc 








LU i-i 


O >- 




>- 




00 Z 


u 


1 


1 




z 


o 


z 


o z 




3 < 


I- z 


X 


Z X 


3 


O -J 


3 -J . 


u3 




Q. 


U OL ' 




O 


Q. O 



o o . o o 

o o ci o 

• . t f f • 

^ o o in 



I t I I } 



> 

LU 
O 



< 
X 



to 
z 



00 

o 
u 



o 

LU 

00 K 

< o 
X o 
u u. 

3 a 

cuoo 



CO 

o 

09 

LU 



00 Q. 
O 3 

uj a 

Z LU 

< 

-J X 

-J < 

LU X 



00 

o 
o 



00 Q. 
O 3 
U 00 



< 



0 


0 




0 0 


0 


0 


0 


0 


0 0 


0 




« 


• 


• • 


• 


0-4 




0 


0 0 


in 




rg 




•4- 0 


rg 




r 




rg in 


0 










in 












1 




1 


1 t 


1 


_| 










-< 




LU 






3 




00 


^ -J 




Z 


X 


oc 


< < 




Z 


0 


3 


3 3 


00 


< ' 0 Z Z 0 






JUt^ z 


u 








< < 




>- 


0 


z 
















< 






>• >• 


2: 


-J 






oc OC 


3 


< 




LU 


< < 




00 


> 


> 








< 


•< 


< < 


Z 


h- 


oc 


? 


00 00 


LU 


Z 


h- 






X 


LU 






oc Z 


Ok. 




Q 




K 




3 


3 


3 


00 X 


3 


h- 






Z Q 


Of 


00 


00 


00 




LU 



ERIC 



IV-51 



TAEG REPORJ NO. 12-3 



o o 

0 0 

0 ^ 

1 I 

< ^ 

fr« o 

3 

o *^ 
o ^ 



o 

u 



>0 GO ^ ^ rg ^ ^ 

o o ' ^ ^ ^ 



*=4 f=} 



I) 
o 
u 



I I I I I t I I I I I r r 









-J 






1 t 


to 




Q 




O 










o 


CQ ai 




^ id 






u 








h- 






ce z 


h- 


o 




3 O -J 


a. 


o. u a 










O 



o 

a 

to Q 

oi to fig: 

h- h- < 

< z o 

3 iii CD 
O O 

<o Z 

0^ h- Q 
CD */) 

til 

U. U. O 
O O < 

CC 

O O > 

z z < 



z 

o o 

^ Of a 
H cb: or 

to to 
o a: to 
d. a oc 

K u a. 
z 3 
uj a: z 
o h- »^ 
3 to ac 

K Z Q 
to •-^ < 



liO ill 
O OI u 
CO 

oc < < 
X «J 

to o 
K oc < 
Z 3 > 
UJ GC < 
X 

a. a. a. 

M fM^ kMl} 

3 3 3 

a a or 

tJJ UJ UJ 



U. U. U. UL «J ^ 
O O O O < < 
3 3 

• • • 0 Z Z 

o o o o Z Z 
z z z z < < 



o 
a 
or 

o 

CC Of 
ox C2C 
CP 

QC h- to 
Z Q 3 
CJ O 

uj z: z 
a. to < 
• OI «j 

UJ h- -4 
CC < ^ OI 
< £ Ql. U 

3 a to 

Of 0 3 i«« 

«^ QC to a: 

0 c/) • 0 

o z o o 
z z z 



ujuMUjuiujiuiJUkyujujujujuj 



IV-52 



1.3 



TAEG" REPORT NO. 12-3 



2: 2 
LU UJ 
U U 

UJ UJ 

o o 



in 



a: 

UJ 

>- 



o o m 

o r\j in 00 f\j 

o m in m 



0 0 0 0 0 0 • 

<^ 00 o o o 

<\j (\j a> 

in fTi rn 



0 ^ 

1 t 



UJ 

K < 
< 

K o 

2 

3 I- 
O < 
U -J 

to UL 

Q 



>- 



ct: 

UJ 



o >o r\j r\j 
o fT) m 
o fO tn 



000 0090 

00 no o f** 
r\j CD »^ 0^ 
:n ^ >r 



0 00 0 0 0 

O rsj r\j no 

o m ro ^ 

o fo in m 



0 • • e e 

O.-^* C>J o ^ 

00 >o o 

PsJ 00 0^ ^ 

in ^ <<r 



1/) 



< 
UJ 

> 



o in o 

o 0^ »-* rsi 

o rsj in in 

r\j CM 



^ in 



^ o 

00 r\j o m m 

r\J O >f OD CD 

tn CM 



o 
u 

uu 
I/) 

O 
U 



UJ 

> 



o 
o 
o 
o 



• 0 0 0 0 Q 0 

in ^ rn m vo 



(M in CM 

CM '4' ^ 

o 



CM 00 00 00 
CM fM CM F«i 



o 

o ^ 
^ >o 
o o 



00 





ft 






to 




2^ 






UJ 00 


-J 


Q 


to 








to 




1 i 


UJ 


^<-« 




or: 


to Q 


0 




UJ to 


0 




K K < 


C£. 


< 


< z 0 


UJ QC 


u 


3 UJ CO 


UJ 




Q Q 


s: CL 




3 Z 




X 


q: K 0 


z 0 


u 


0 to 


z 


UJ 


UJ 


uu 1^ 


»- 


li. a. 0 


to 2 




0 0 < 


•• CSS 2: 




oc 


K 3 < 




0 ♦ uu 


o 0 -J 


Q. 


0 0 > 


0. U Ql 




z z <t 








0 





Q 

Of 



o 

O Q Q 

o: Q 3 
a: a: 

to 

to to 

O QC K 
H uu Z 
U Ql UU 
3 X 
CK.Z 0- 



tO UU 

uu ^ 
to CO 

< < 

X ^ 

u 

q: < 

3 > 
Q- < 

Ql Ql 



UO X 3 3 3 

z Q o or o 

< uu uw uu 

u. a. li. ^ _j 
o o o < < 
3 3 

« 0 0 

o o o z z 

Z Z-Z < c;Jl 



X 

Q K 
Of Z 

< 

K X 

UU 
UU 

o. < 
X 

uu 

Qt: a: 
< K 
3 to 
o z 

to 



Q 

Of 
QC 

to 

Q 3 
O O 

Qt: uu 
z 
to < 
uu ^ 
i-a -J 
^ UU 

a u 
a to 
3 H-» 

to X 



0000 
z z z z 



IV-53 



TAEG REPORT NO, 12-3 



s: ^ 

o u 

ca: 

a. OL 

o o 

G 0 

o ^ 



I t 
o 



0000000 0000 

ooooooopooooo 



LU 

> 



UJ 

> 



> 



0000.000 0000 

OOOOOOOOOOOOO 



0000000 OOOf 

O C\J fSJ F=i CD O <f ^ 



4/) 



> 



aooeooo oo.ot 
O O GO ^OO^OOGOflO 

OCf'^<0*=^ CO OOEAomin 

00 €0 ff^ 00 CD 



O 

a 

a 
u 



















t 1 


HI 


O 




O 


«J 




U 


CO UJ 


1^ 


sc & 


Z 




X 




o 










(/) z 




eg Z 




3 « 




Z> O «J 




a. u Q. 










O 



> 



• OoOOOO OOOf 



Q 

ca 

a: z 
to 

Ui (/) 
H >- 

< Z 
3 UJ 
Q O 

< D 

U. 

O O 



O O 



00 

z 

o o 
oc 

l/> CO 

o a: 
o & o 

< H U 

o ;^ 3 
oo Ui a: 

Q H 
Z 3 00 
O H Z 
»-» 

Ui 

o u. u. 

< aa 
ec 

Ui • • 

> o o 

<s z z 



00 

O O UJ 
or 00 

o: QC < 

X 

00 00 u 

H as: 

UJ Z 3 
OL Ui Ol 
X 

Z Ol OL 

X 3 3 
O or 
< Ui Ui 

U. U. «J 
O O < 

3 

) 4 Z 

o o z 
z z < 



UJ 
CO 

< o 

^ a 

> H 

< Ui 

Ui 
CL U. 

3 Ui 

a cc 

Ui < 

< 00 



o 
a 

OL 

X 



< o 

X or 
a: 

< 00 

a: Ui 

ct ^ 
H a. 

</) OL 
Z 3 

M to 



o 
d 

00 
3 

a 

Ui 

z 



Ui 

u 

00 



Z • 0 9 

z o o o 
< z z z 



0 

o 



IV-54 



TAEG REPORT NO. 12-3 



(-0 

>• 

< 



00 

o 
u 

UJ 

(/) 
O 

u 



UJ UJ 
U O 
CC OC 
UJ UJ 

O O 
« f 

O sO 



I I 

UJ 
UJ H 
I- < 
< 

z 

H O 
Z 

=> K 
O < 
U -J 

z 

O 1^ 



Q 




• 


• 


0 




• 


• 


0 


0 


0 








UJ 




o 




O 


sO 


m 




m 


O 


O 


m 








o 




O 




m 


<r 


CD 


CD 




nO 


m 


(7^ 


in 


z 






GO 


o 


>o 






o 






0 


• 


• 


Z) 


Z CNJ 


o 


>o 


in 


o 




CNJ 








CNJ 


in 


(7^ 


o 


UJ >t 


in 


on 


in 




00 










CD 


>o 


m 


u 


CC o 




















O 


>o 




oo 


























in 




^'•^ 


















in 








Q 






























t 


• 


• 


it 




• 


• 


• 


• 


• 









♦^^CDCNjsOrMOCOO 
O cDsj-mcDr^O'-MO 
i^Hr^cNjNO^^ooit' 
»^Z^C3(7»cNjcNjcomr*r^ 

QUjNtCNjNONt(\iO»-tCNJ 

ica^ONOcDsor**-* 
ZarooNffn ^ 

O 3 sf CNJ 

Z u 



CNJ 

(7^ 
m 

in 



00 



O v4" (7» CNJ O 

ooZ'-^inrHf^-sfp^ 
i-4<>o>o*^*r^cNjo 
QHroin^^fOoo 
I 00 CD in GO 

Z Z GO (7* rO r-t 

o o m 
z o 



Csi o 

in 

CNJ 

o 



o 

CNJ 

m 
o 

in 



^ 00 ^ 

GO rn 

0 0 0 

>J- ^ ^ 

in rn GO 
(7* (7* 

CNJ 



Q 




















0 








UJ 




o 






o 




CNJ 


m 


o o 


CNJ 


in 




>r 








(7* 


(7^ 








(7* 


GO 


fO 


(7^ 


m 


(7^ 


Z 


Z 


CNJ 




c\» 


CNJ 


in 




O 


o 


o 


0 




• 




< 


CD 


ro 




in 


vO 




f— * 


(7^ 


in 


m 


CM 




O 






(7* 


in 




00 




*-( 


in 


o 


>o 


in 


in 


U 


UO 


o 


m 


CNJ 


m 


nO 






vO 




in 


in 


in 


00 


Z 


00 




CNJ 










GO 






CNJ 


>— « 


o 


CNJ 
























Q 


u 



























O CNJ 

r* CNJ 0^ 

f) 0 0 

1^ in r* 
in >o 
r- fh 



z 

UJ 00 

00 

} I 

Q 

o 
oc ^ 

UJ. 0^ 
00 UJ 
3C Q. 

z o 
z 

UJ 

00 Z 
0^ z 
H => < 

o ^ 

Ql U Q- 

z 















00 






























> 






< 






























< 






0^ 


















LU 






to 




>- 


< 














00 


a: 








< 




< 


•J 


UJ 


< 












z 




1*^ 


-J -J 


UJ 










o 


00 




UJ < 


> 








00 




UJ 


< 


> 00 








< 


z> 




0^ 




< 








z 


o 


< 




< 


a: CC 


< 




00 


o 


UJ 


a: 




00 


H O 


0^ 




lU 




z 


UJ 


00 








Z 




h- 00 


< 


Q. 


o 




H U 


00 


UJ 




U UJ 




O 


u 


Z 


Z => 








z> 








UJ 


UJ 0^ 


z 


a. 




oc ^ 


uu 






o 


Q H 








H- a 


u 


< 






3 00 




z> 


U 


00 CL 


00 




Q. 




H Z 


Q 




< 


z z> 




o 




00 


00 »-H 


< 


UJ 


LL 


00 


a: 





z 
o 



CL 00 

z o 



< z z 

3 uu UJ 
Q Q Q 

< 3 3 
0^ H H 

O 00 00 

0^ 

OJ UJ UJ 

a. Q. a. 



I/) 00 00 

o o o 
u u u 



IV~55 



TAEG REPORT NO. 12-3 



a: 2 

u u 

uj ai 
a. a. 

o o 



rsjinoo^oo^orgoGOGooir^ 
rnoo<\ifwcoo^o o 

>• Cy\ O ^ CO 



0 ^o 

1 I 

UJ 
UJ K 
N» «^ 

z 

2r 

O < 
U ^ 

I/) 

*«• Z 
Q 



< 



00 

m m GO 



0Dc^^^-o•^otf^ vO 

o c> fv 00 cNj 

00 1-1 CO m ^ ^ O 

fo rg in ^ >o 



C\J»-<'<ftf\Oh-OC>O^f*-0^G0 

ooo^^h-o^om vO m m 00 
O^<>rgooin^h- rsjh-h-h- 

oor^*-<ooin<^ o i-h 

<^ OsJ in i«l r«4 

^ CO o 



00 

> 
< 

O 

o 

UJ 
00 



2: 

UJ CO 



< 
UJ 
>- 



00 

o < 

U UJ 

> 

u 

00 

o 

I 

z 
o 
z 



00 

i 1 


• f 

00 


o 




o 


3 


cc ^ 




Uj a: 


UJ 


CO UJ 




x: a. 








z o 


t/5 


z 


O 


UJ ¥^ 


u 


00 z 




«« Q£ Z 




^ 3 < 


3 


3 O U 




O. U CL 


H 


z 


3 




O 



o>h-.-<friorgoinofnGoom 

vOrvjvOooOvO^m CD 00 
oomcrrnyOinNco 

•.^ 00 ««4 r-H o 

in c\j OS 



0^ vO >o #^ o >» o 
m ro >o vo in o Q 

m fO (Tl >5) r-s} 



00 

UJ 

cz: UJ 
> 

00 

z z 

UJ Ol 

o o 

3 3 

00 LO 



< 
> < 
<i 

^ UJ 

< > 

to I— I 

o a: K 

H- Z 

U 00 UJ 

3 ac 
z a 

^ 0<"4 

00 Z 3 

z a or 

1*4 < UJ 



00 

a: 

UJ 



-J 00 

< 3 

Z O 

00 O Ol 

UJ z 

00 < 
U UJ ^ 
3 t-i ^ 
«J «J UJ 

a, o 
u 00 a 00 
z 3 i-< 

u> u) ac 



o in a* 

00 

00 00 rn 
00 o 

CO o rg 

0> 



Z 

o 



^ Ol 

z < 

O 3 

o 

K < 

< 

OC O 
UJ 

Ot 

O Ol 
Ol 



3 

z o 
a. 



z z 

UJ UJ 

o a 

3 3 

00 00 

€C OC 
UJ UJ 

a a 



K CO O^ CO 

o o o o 
o u u 



IV-56 



TAEG REPORT NO. 12-3 



a: 2 

LU LU 
U U 

ac a: 

LJJ LU 

a. 

b o 

• e 

O 



f I 

LU 
LU H 
H < 
< 0^ 

H O 
Z 

O < 
<J -I 
C/) U. 

Q »-» 



0 

O 



oooooototoot 

oooooooooooo 



< 

IJJ 



0^ 



< 



GO 



< 

yj 



OOOOOOOOOOOOO 



0 • • 0 • 

c>rsj>5-inoco„^ 

c^roooicornooD 

oooomofNJrgrgrn 
O 00 o '/> I m 
rsi ^ rsi 



• • 9 • S 0 

O rg o r*l 

O qo -4 
^ kn 

o ir» 

o ^ 
in 



I 



> 

O 
U 

tu 

00 

o 
u 



00 

o 
u 

u 

00 

o 
I 

o 







z 




z 






LU GO 






00 

1 i 


TS: 




o 






o 


3 




QC <^ 


00 




LU Q£ 


LU 




03 LU 


QC 




X Q. 








H 






00 




z 


O 




LU t-4 


o 




00 Z 






cx: z 




H 


3 < 


3 


3 O -1 


O. 


o. 


U O. 


H 


z 




3 






O 



< 

LU 
> 



QC 
< 
LU 
> 



tnoofOoo>Ofr\fNjir\ 
»^ sf in 00 00 

nO CO (Nl 



O p-4 rsj 00 
>o >,tn 
00 

sr *o 

00 

o 



^oo«-<ooo^o<^ 
r*-- in m fNj o o o 
Nt^^h-oooofNjNtrg 

o o fM in in c> c?* 
^ ^ m 



O stj O Psl o 

sO so ^^• '.i* 

^ 00 

00 ^ 

00 



00 
LU 



z 
o 

3 

Q. 00 

z o 

U LU 

< H H H 

z < z z 

O 3 LU LU 
00 Q Q Q 
3 H < 3 3 
O <r H H 

LUQ^OOOOO 

OOHOqCHLUi-^ ZLU 

H*HZ>-«H00<Q-a:QCQ^ 

>-hcjooluhulu^Olululu 

ZZ3»-^X»-<3h^_J Q.Q.Q. 
LULUQCZQ._ja_|LUU 

33i/>X3U00Q-00H 00.00 o0 
HHZQO<<Z3«-iOOOO 

oooo»»h<luuu»^oos:huuu 



< 

a: to 

u U LU 
LU < >u 
<t > 00 >«i 
-I < H 
< QC QC < 



00 

QC 



< 
z 

00 O 



ERIC 



IV-57 



TAEG REPORT NO. 12-3 



JOB COST MODEL 

• Description of BIlTet Cost Model* 
i Billet Cost Model Variables 



lat^Jn^^fJlT documentation on the Navy Military Manpower Billet Cost 
O Personnel pro^^^ded by the Bureau of Naval 
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DESCRIPTION OF BILLET COST WODEL 

INTRODUCTION. The Chief of Naval Personnel has developed the data in this pub- 
lication for use in computing the life-cycle 'costs for major Navy weapon sys- 
tems (ships, aircraft, etc.). Yhese data are provided to civilian contractors' 
Competing for and/or participating in the formulation of weapon system design 
and procurement,^ and are used by the Navy and the Office of the Secretary of 
Defense in the evaluation of contractor proposals. In addition, they are used 
for many other management purposes, wherever the projected costs of military 
manpower are significant. 

Manpower billet costs, as defined by the DOD Manpower Cost Model Study Group, 
include the total cost to the U.S. Government of manning an established opera- 
tional military billet, or a mili tary- bil let which would be established within 
a proposed system. These costs are presented as annual costs for both officer 
and enlisted personnel and are used in costing the life-cycle of the system 
under consideration. Manpower billet costs differ significantly from appro- 
priation-oriented personnel costs which are used for budget planning purposes, 
since they contain values whicji are not taken into, account in appropriation- 
oriented costs. For example, the values established within the formula for 
the purpose of costing the "down time" of personnel in service schools, and 
the values developed to reflect the effect of "continuance rates" are not con- 
tained in personnel cost data used for budget plahnning purposes. 

Manpower billet cost data contained in this publication are not complete costs, 
but reflect mainly the cost to the Navy and the Department of Defense. 

Manpower Billet Costs are developed for both officer and enlisted personnel. 
The costs are computed in a manner which results in an "operational billet cost 
per year" for all operational billets in the Navy with the exception of flag 
billets and paygrade E-1 billets. In order to compute these costs, the Chief 
of Naval PersonneThas developed a Billet Cost Model. Basic pay data included 
in this publication are based on January 1973 Pay Tables. A summary descrip- 
tion of the major cost elements within the Billet Cost Model is contained in 
Section III. This general desription is applicable to both officer and enlist- 
ed billets, and is provided as an explanation of billet costing. It is 
presented in terms of enlisted billets to simplify the explanation. 

It is planned to revise the publication annually to include changes in the 
cost elements brought about by military pay raises, changes in training costs, 
etc. 

DISCUSSIOM. Manpower costs are accrued from operational billets. An opera- 
tional billet is best described as a job position which is being filled at all 
times by personnel in an operational status. All other personnel such as 
students, patients, prisoners, or personnel in a transient status are* not 
occupying operational billets. Manpower related costs incurred during opera- 
tional times are consideredo'n the model formulae as "UP" costs, and costs 
incurred during nonoperational tim§s are known as "DOWN" costs. DOWN costs 
include pay related cost, (when i^ncurred in the DOWN status) and an apportion- 
ment of support and certain one-£ime occurring costs, such as procurement ^ 
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(recruiting) costs » based upon the length of the DOWN timef The concept is 
that DOWN costs represent a manpower resource investment cost which the Navy 
must make in order to obtain future operational service.- Therefore, all DOWN 
costs are amortized, over future operational billet years. 

One m^j^r elemant of cost to the U.S. Government which must be apportioned to 
operational billets is that related to retirement costs which occur after 
career men finish their operational duty. The total cost of retirement per 
individual (called the principal value of retirement) is developed on the 
basis of life expectancy datav^ This cost, computed on the basis of average 
age at the time of retirement from specific grades, is distributed by the 
model back (in time) over the operational billet years. Several methods of ' 
distributing this cost have been developed and tested. The one found to be 
the best for billet costing purposes and used in this model, is the method 
which distributes the principal value of retirement as a fixed percentage of 
active base pay. This method was approved for billet costing by CNO, 
0P-90Ell/pas Ser 620P90 of 22 April 1969. 

A primary factor characterizing manpower bi 1 let , costs is that these data are 
to reflect the total cost to the U.S. Government, not just to the Navy or the 
Department of Defense, of manning an established operational military billet, 
or a military billet which would be established within a proposed system. 
The costs developed by the model results in an "opfe rational billet cost per 
year" for all operational billets in the Navy except flag, billets and paygrade 
E-1 billets. 

\ 

In developing life-cycle costs, the Model views the manpower picture as a 
flow of men through each rating as well as through each billet. They are 
procured,^^' training, utilized and, as time passes, they are lost through non- 
reenl jstment , death, retirement or other attrition factors. Manpower costs 
are thus computed from initial procurement to the end of retirement and 
charged to an active duty base of twenty-five operational billet years. 

Each rating is viewed by the Model as a flow of men through a pipeline. The 
pipeline 1s divided into year-long intervals and all costs Incurred by the 
rating in each interval are noted. During, the interval, some men are lost 
through various leaks in the pipeline, e.g., nonreenlistment , change in rating, 
death, etc. No attempt is made to analyze these leaks. The only criterion of 
consideration is the quant>ty of men entering the next interval. The relative 
number of men is considered because the actual number affects only the total 
cost and not the cost per man/billet. Accordingly, the flow is normalized ^ 
such that one man retires at the end of the career span, i.e., twenty-five 
years. Thus, normalization is a function of the flow rate in a given rating 
and varies with the different ratings. It dictates that the number of men 
which must be introduced into the pipeline the first year to produce one man 
retiring at the end fluctuates significantly. 

In surmiaryp the Billet Cost Model is quite flexible. It accounts for the many 
variables which make up life-cycle operations costs. It determines which cost 
element applies to all ratings by year of service or grade or which cost 
element applies to a particular rating. As each element is identified by a 
card in the model, a value 1s added to each appropriate rating packet as 
constant $ by year or constant $ by grade. A detailed list of cost items 
prese'ntly being used is shown in the Billet Cost Model Data Summary. 
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The preliminary computations to obtain the input data' cards for the model are 
external' to the model. The model will accept numbers up to six digits. The 
precomputations and source- of data have been kept as a matter of routine so * 
that future cost changes' can^t^e incorporated on an "as-requi re'B" or "as-occurred" 
• basis. Moreover, there is^j| fixed number of rating packets that must be run 
at one timfe. This allows fof experimentation with a single rating packet or 
a group of rating packets. A -hypotheti cal rating can be created by defining 
a new rating XXX and' preparing a cost/statistical profile the'same as a regu- 
lar rating. The hypothetical rating can be run by itself or with a regular 
rating to obtain a comparison of results. Similarly, either the enlisted or 
officer decks can be run separately. 

MANPOWER BILLET COST MODEL DATA DESCRIPTION. This section contains a brief Sum- ■ 
mary description of some representative cost, elements within tjie Billet Cost Model. 
While it is not a complete list, it indicates the range of data usedc-to develop 
billet costs. Since actual values are subject to significant change, only gen- 
eral data are given. 

Grade Costs 

Sources: Budget FY 1972 Military Personnel, Navy 

Budget FY 1972 Operations & Maintenance, Navy 

These costs include the following, computed for each officer and enlisted pay 
grade: 

Family Separation Allowance 

"Q'' Allotment 

Quarters Allowance BAQ 

Sea and Foreign Duty Pay 

Tuition Assistance aad Dependent Schools. 

Annual Costs 

Sources: Budget FY 1972 Military Personnel, Navy ' 

SECNAV Task Force on Navy/Marine Corps Personnel 
Retention, Vol . X 

Includes annual- costs such as: ^ 

Clothing Allowance 
Death Gratuity 
Commissary 
Exchange 
Medical Costs 

Subsistence Allowance, etc. 
Base Pay . ^ 

Source: DOD Military Pay and Allowances Entitlement Maaual (as changed). 

Hazardous Duty Pay (Aviation and Submarine ) 

Source: DOD Military Pay and Allowances Entitlements Manual. 
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Continuance 'Rates 
; Source: BuPers Master ^rsonnel Tapes. 
Averafge' Length of Service to Advance 

Saurce: Pers-A12 for enlisted * . 

Pefs-All for officers . ^ 

Constant Dollar Cost by Grade • • 

Sources: Navy and Marine Corps Military Personnel Statistics, NAVPERS 'l5658 
- Proficiency Pay List, Report E-T/O » 
• Manual of Navy En 1 is ted Classifications , NAVPERS 15105 
Budget 1972 Military Personnel, Navy 

Includes cost items that can be identified by- Rating' and Grade, such as: 

Proficiency Pay ^, 
. ^ ' Hazardous Duty Pay (non-crew) 
Flight Deck Duty Pay. 

I 

Transportation Costs • % 

Source: Budget FY 1972 Military Personnel, Navy 

These costs are divided into four categories: accession travel, change of sta 
v.^^tfon travel, training travel and separation travel. Computations are done sep 
arately for enlisted and officer personnel and data are included only where ao 
propriate. ^ ^ 

Constant Dollar Cost by Year of Service 

. /v Sources: DOD Military Pay and Allowances Entitlements Manual 
BuPers Instruction 1133. 18C 

Pers-'A12 . ■ ' 

Includes cost items that can be identified by Rating and Grade and Year of • 
Service, such as |^eenl istment Bonus and Continuation Pay. — ; 

Median Length of Service , 

■ :p , ■ ■ 

Sources: BuPers Report E;159H Report of Enlisted Personnel on Active Dut^' 
Pers~All 

Officer Management Si/nulation Model 
School Training Costs 

Sources: Computed by Occupational Standards Division. Naval Personnel Prodram 
Support Activity. Washington, D.C., for enlisted. 
For officers, data procured from Pers-All and 0P-992E. 

^"nI^^®n/]^•^?'^'/^^^•^^ *° ^^^^^^^ training, including procurement costs as 
a part of initial training. 
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DECISION TREE EVALUATOR 

Tree Format With Nodes and Variable Designators 
ETAM Decision Tree Format 

Program Listing 

Run CTREE 

Run Command 

Gamma Command 

List Command 

Gamma Command 
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ETAIW DECISION TREE FORMAT* 



:l C: ETAM DECISION TREE FORMAT 

.2 GENERATION 

• .3 TYPE PR(2).TE(2), DE(2) 

.4 NODE DECIDE, DE 

.5 VAL(1)=6HDECIDE 

•6 IVAL(2)=1 

.7 NEXT NORISK 

.8 NEXT RKPKGl .COSTl 

.9 END 

1. NODE NORISK, DE 

1.1 VAL(1)=6HN0RISK 

1.2 IVAL(2)=2 
1-3 NEXT ACPTOO 
1-4 NEXT -,D 

1.5 END 

1 .6 NODE ACPTOO, PR 

1.7 VAL(1)=HACPT00 

1.8 IVAL (2)=3 

1.9 NEXT IMPLOO,PU 

2. NEXT -,1.-PU,C 

2.1 END 

2.2 NODE IMPLOO.PR 

2.3 VAL(1)=6HIMPL00 
^2.4 IVAL(2)=4 

2.5 ., NEXT -,PV,A 

2.6 NEXT -,1.-PV,B 

2.7 END 

2.8 NODE RKPKGl, PR 

2.9 VAL(1)=6HRKPKG1 

3. IVAL{2)=5 

3.1 NEXT PSUCCl ,PI1 

3.2 NEXT PFAILl ,1.-PI1 

3.3 END 

3.4 NODE PSUCCl ,DE 

3.5 VAL(1)=6HPSUCC1 

3.6 IVAL (2)=6 

3.7 NEXT ACPTll 
3.8. NEXT -,D 
3.9 END 

4. NODE ACPTll, PR 

4.1 VAL(1)=6HACPT11 

4.2 IVAL(2)=7 

4.3 NEXT IMPLll ,PW1 

4.4 NEXT -,1.-PW1 ,C 

4.5 END 

4.6 NODE IMPLll, PR 

4.7 VAL(1)=6HIMPL11 

4.8 IVAL(2)=8 

4.9 NEXT -,PX1,A 

5. NEXT -,1.-PX1,B 
5.1 END 



ERIC 



•Developed using SRI (Stanford Research Institute) tree language 
system, SRI, Menio Park, California 94025. 
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5.2 NODE PFAIL1,DE 

5.3 VAL (1)=6HPFAIL1 

5.4 IVAl (2)=9 

5.5 NEXT ACPT10 

5.6 NEXT -,C o 

5.7 END 

5.8 NODE ACPT10,PR ^ 

5.9 VAL(1)=6HACPT10 

6. IVAL (2)=10 

6.1 NEXT IMPL10,PY1 

6.2 NEXT - J .-PYl ,C. 

6.3 END 

6.4 NODE IMPL10,PR 

6.5 VAL(1)=6HIMPL10 

6.6 IVAL(2)=11 

6.7 NEXT. -,PZ1,A 

6.8 NEXT -,1.-PZ1 ,B 

6.9 END 

7. C: INPUT STATEMENTS 

7.1 .WRITE (5,84) 

7.2 READ(5,85)A 

7.3 READ 5,85)B 

7.4 READ 5,85) C, 

7.5 READ(5,85)D 

7.6 READ 5,85}C0ST1 

7.7 READ(5,85)PI1 
7.3 READ(5,85)PU 
7.9 READ(5,85)PV 

8. READ 5,85 PWl 

8.1 READ(5,85)PX1 

8.2 READ(5,85)PY1 

8.3 READ(5,85)PZ1 

8.4 84 FORMAT (46H ENTER ABC 

8.5 85 FORMAT (F8.2) 

8.6 START DECIDE 

8.7 END 



D COSTl PIl PU PV PWl PXl PYl PZl) 
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.RUN CTREE(31,76) 

* DECISION TREE EVALUATION SYSTEM * 

* PREPARED FOR ARPA * 

* BY THE SRI DECISION ANALYSIS GROUP * 



TREE LANGUAGE FILE: NAVAL 

FORTRAN FILErFNORT 
FORTRAN: NF0RT.F4 
LOADING 

NFORT 8K CORE 
EXECUTION 

TREE FILE: NTREE 

ENTER A B C D COSTl PIT PU PV PWl PXl PYl PZl 

28.0 
-3.8 
-.5 
0.0 

-n.1 

.95 
.75 
.50 
.85 
.90 
.75 

.50 ■ ■ " .. 
TREE FILE:NTREE 

TREE HAS ^ n NODES 
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COMMAND: RUN 



LOTTERY , ACTIVE NODE PRINTOUTS, STARTING NODE, OK ?YES 



CERTAIN EQUIV. OF STARTING NODE: 9.3183998 
SELECTED DECISION: RKPKG ALTERNATIVE 2 NODE NO. 
ANALYSIS OF ALTERNATIVES 

NAME ALT. NODE NO. CERTAIN EQUIVALENT 
NORIS 1 3 8.9500000 

, RKPKG 2 10 9.3183998 



10 



SUCCESSOR 

NODE NODE DEC- NODE NODE ROLLBACK 
NO. NAME IS I ON NO. NAME VALUE 



11 
10 
9 
8 
7 
6 
5 
4 



DECID 
RKPKG 
PFAIL 
ACPTl 
IMPLl 
PSUCC 
ACPTl 
IMPLl 



10 
8 



RKPKG 
ACPTl 

ACPTl 



9.318400 
20.41840 
8.950000 
9.950000 
12.10000 
21 .02200 
21 .02200 
24.82000 
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LOTTERY ON ALL REWARDS IN TREE. 



LOnERY.MIN= -14.900000 
MEAN= 9.3184000 50= 



MAX= 16.900000 
13.001403 



LOTTERY 
VALUE 



PROBABILITY CUMULATIVE 



•14.90000 
■11.60000 
16.90000 



0.09950 
0.15500 
0.74550 



0.09950 
0.25450 
1.00000 



1.0000- + 
+ 
+ 
+ 
+ 

0.8000- 
+ 
+ 

+ 
+ 

0.6000- 
+ 

+ ' 
+ ■ 
+ 

0.4000- 
+ 

■• ■ + 
+ 

+ + 

0.2000- 

+ . 
. + 

+ • + 

+ 

0.0000- 

+ , +++++++++ , +++++++++ , +++++++++ , +++++++++ , I I I I I I I 1 + , H 11111++ , ++ 
-22.50Q -15.000 -7.500 0.000 7.500 15.000 22.500 
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COMMAND: GAMMA 



RISK AVERSION C0EF.=.02 



COMMAND: RUN 



/ 



GAMMA= 0.20000000E-01 , LOTTERY , ACTIVE NODE PRINTOUTS, STARTING NODE, OK ?YES 



CERTAIN EQUIV. OF STARTING NODE: 7.4712394 
SELECTED DECISION: RKPKG ALTERNATIVE .2 NODE NO, 
ANALYSIS OF ALTERNATIVES 

•NAME ALT. NODE NO. CERTAIN EQUIVALENT 
NORIS 1 3 6.8869030 

RKPKG 2 10 7.4712394 



10 



SUCCESSOR 

NODE NODE DEC- NODE NODE ROLLBACK 
NO. NAME IS ION NO. NAME VALUE 



11 
10 
9 
8 

7 
6 
5 
4 



DECID 
RKPKG 
PFAIL 
ACPTl 
IMPLl 
PSUCC 
ACPTl 
IMPLl 



10 RKPKG 
8 ACPTl 

5 ACPTl 



7.471239 
18.57124 
6.886903 
6.886903 
9.613395 
19.26884 
19.26884 
28.74201 
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COMMAND :UST 



ERIC 



FORWARD OR BACKWARD: FORWARD 
NODE DECID NO. • 11 TYPE -DEC 

ATTR: 1.0000000 

SUCCESSOR 3 

SUCCESSOR 10 ATTR: -11.100000 

NODE RKPKG NO. * 10 TYPE CHA 
ATTR: 5.0000000 
SUCCESSOR 6 ATTR: 0.95000000 
SUCCESSOR 9 ATTR: 0.49999997E-01 



9 TYPE DEC 



NODE PFAIL NO.^ ■ 
ATTR: 9.0000000 
SUCCESSOR 8 

SUCCESSOR 0 ATTR: -0.50000000 



8 TYPE CHA 



NODE ACPTl NO. 
ATTR: 10.000000 
SUCCESSOR 7 ATTR: 0.75000000 
SUCCESSOR 0 ATTR: 0.250D0000 



7 TYPE CHA 



NODE IMPLl NO. 
ATTR:' 11.000000 
SUCCESSOR 0 ATTR: 0.50000000 
SUCCESSOR 0 ATTR: 0.50000000 



NODE PSUCC NO. ' 
ATTR: 6.0000000 
SUCCESSOR 5 
SUCCESSOR 0 ATTR 



6 TYPE DEC 



0.0000000 



NODE ACPTl NO. 5 TYPE CHA 
ATTR: 7.0000000 
SUCCESSOR 4 ATTR: 0.85000000 
SUCCESSOR 0 ATTR: 0.15000000 



4 TYPE CHA 



NODE IMPLl 
ATTR: 8.0000000 
SUCCESSOR 0 ATTR: 0.90000000 
SUCCESSOR 0 ATTR: 0.10000000 



3 TYPE DEC 



NODE NORIS NO. 
ATTR: 2.0000000 
SUCCESSOR 2 
SUCCES^Of^ 0 ATTR: 0.0000000 



2 TYPE CHA 



NODE ACPTO NOX 

ATTR: 3.00000bo 

SUCCESSOR 1 ATTR; 0.75000000 
„J^CCESSOR 0 AnR: Q. 250000000 
NODE IMPLO NO. 1 TYPE CHA 

ATTR: 4.0000000 

SUCCESSOR 0 ATTR: -0.50000000 

SUCCESSOR 0 ATTR: 0.50000000 
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-0.50000000 



28.000000 
-3.8000000 



■0.50000000 



28.000000 
-3.800000 



-0.50000000 

28.000000 
-3.8000000 
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COMMAND: GAMMA 

RISK AVERSION COEF.=0 

t 

COMMAND: RUN 

LOTTERY , ACTIVE NODE PRINTOUTS, STARTING NODE, 0K7YES 

CERTAIN EQUIV. OF STARTING NODE: 9.3193998 

SELECTED DECISION: ' RKPKG ALTERNATIVE 2 NODE NO. 10 

ANALYSIS OF ALTERNATIVES 

NAME ALT. NODE NQ. CERTAIN EQUIVALENT 

NORIS 1 .3 8.9500000 

RKPKG 2 10 9.3183998 ' 



SUCCESSOR 

NODE NODE DEC- ' NODE NODE ROLLBACK 
NO. NAME ISION "NO. =NAME VALUE 



11 
10 
9 
8 
7 
6 
5 
4 



DECID 
RKPKG 
PFAIL 
ACPTl 
IMPLl 
PSUCC 
ACPTl 
IMPLl 



10 
8 



RKPKG 
ACPTl 

ACPTl 



9.318400 
20.41840 
9.950000 
8.950000 
12.10000 
21.02200 
21 .02200 
24.82000 
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SECTION V 
SCENARIO EXAMPLES 



This section contains examples of applying ETAM to all or part of the 
recomrnended process for evaluating a training innovation. The first scenario 
takes a proposed innovation from the innovator's description through the 
series of ETAM tasks. The results of each task are presented as a general, 
but not complete, model for. the analytic and decision process. The remaining 
scenarios are treatments of subsets of the assessment process which were 
felt to require further elaboration. Also, some sample innovations have 
been included to which ETAM procedures might be applied. 

SCENARIO NUMBER ONE - 3-D PROCEDURAL TRAINER 

The innovator experimented with a 3-D simulator for replacing actual opera- 
tional' laboratory training equipment used in certain training tasks. The 
innovator's description of his preliminary work is presented as the starting 
point for the scenario. The reviewer should be aware of the following as- 
pects of the approach to presenting* this scenario materiaU 

1. ' Inconsistencies in the Task 1 description are deliberately introduced 

to simulate realisms^. Actual people don't always see the real versus 
fancied benefit or li abi li ty of the product, even when the data stare 
them in the face. This is one of the values of formal ETAM which sheds 
light on all of it. 

2. The present scenario is an abstract of what might, in real life, be a 
lot of text. But it should be evident that text, other than the context 
of what appears here, would be superfluous and even obscuring. 

3. The major kind of data that might have been added would have been^his- 
torical field experiences showing how poorly most people do on proce- 
dures, and the high error rates almost universally associated with 
procedural tasks. ^ 

INNOVATOR'S DESCRIPTION. The innovator prepared the following write-up aT 
an input to the analysis and decision process. 

EXHIBIT: 3-D SIMULATION OF EQUIPMENT TO TEACH EQUIPMENT PROCEDURES 
Description ' ' ) 

On-the-job evaluation of operators and maintenance personnel shows a 
general inadequacy in the performance of routinized procedures such 
as turn on, calibration, checkout, operation, and so forth. This 
leads to inefficient performance and many errors. 

But the use of actual equipment in training exercises is expensive in 
many ways. The cost of the equipment, plus its maintenance and cali- 
bration for training, limits the number which is practical for training 
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EXHIBIT: 3-D SIMULATION OF EQUIPMENT TO TEACH EQUIPMENT PROCEDURES 
(Continued)... 



purposes. Furthermore, an instructor must usually be present for each 
one, or pair of students working on equipment so that the correctness 
of' each step can be verified, or to provide guidance and interpretation 
when things go wrong, eithpr because of student error or equipment fail- 
3 ure. The combination of these costs may limit student practice on the 
equivalent of hands-on procedure to a few hours, rather than the dozens 
of hours necessary for a reliable proficiency. 

This innovation consists of using a dummy shell of the equipment, but 
with all indicators and controls having an appearance of being opera- 
tional. Controls are manipulable. External test points on which to 
attach test equipment probes are presented and "functional." 

The internals of the mockup consist of circuits which feed impulses to a 
programmed computer. The latter returns equipment indications suitable 
to the control activations. In additjonr, the program generates correc- 
tive feedback to the student. This is presented on a'nearby screen. 
The. screen can present only static images, but it can display a given 
wave shape in response to a given situation and action. 

Several comparative studies had the following results: * ' ' 

1. The experimental group was given the same overall number of lab 
sessions and lab hours as the control group the use of live 
equipment was by the control group. In the experimental group, 
each student had a teaching mockup which he used throughout the 
lab periods, except for the final performance test which was made 
on two pieces of live equipment and monitored by the instructor. 

The experimental group's performance on a scale which combined 
scores on speed and errors; with a high weighting on errors, was 
judged to be 40% superior to the performance of the cdntrol group. 

Retest a month later, showed the experimental group to be 60% 
superior to the control group. These scores suggested an estimated 
reduction in on-the-job training from four weeks to one week in . 
^order to acquire acceptable operational efficiency. 

tests six months after incumbents were on the job showed 30% of 
the experimental group as "outstanding" in performance on the 
subject tasks, whereas only 5% of the control group was judged 
outstanding in their job perfomjance. There were similar superi- 
orities of the experimental over the control subjects at other 
levels of evaluation. ^ 

During training, there was 1% attrition in the experimental group 
and 7% in the control group. 
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EXHIBIT: 3-0 SIMULATION OF EQUIPMENT TO TEACH EQUIPMENT PROCEDURES 
(Continued)... i ' 



2. Cost analysis. Development plus operational costs of the ex- . 
perimental device netted out to $10 per student hour. The costs 
for the actual hardware (control condition) netted out to $9 per 
student hour. These costs were based on a 5-year spread. 

(One liability in the experimental device, ^the operating mockup, 
was the need to run one pass of pilot students through the device 
in actual training conditions in order tb properly debug its opera- 
tion. -For this class, the ordinary complement of actual equip- 
ments had to be available as backup for training while the mockup ^ 
trainers were being debugged.) T 



,.In the control group,' dne- instructor was necessary fpr every four 
students, while in the ipxperimental group, one instructor was* 
sufficient for a class of 20 to 25 students/ 



A large investment would have to be made by the Navy to develop the 
technology for th^ mockup trainer. It could be connected by telephone, 
to a central computer. The ultimate costs for units of this device 
could be drastically reduced if they could be used -In very large- num- 
bers in a large range of job tasks. (The cost analysis figure for the 
device, « cited above as $10, assumed a minimum of a thousand of the de- 
vices in use by the Navy at the end of 5 years. A large investment In 
development funds might have' a good chance of reducing the cost per 
student hour below the $9 stated above.) 

It is, therefore, important to determine^ the widest potential range- 
of-effect for this training function. 

jASK 1 - FORMALIZE THE DESCRIPTION OF THE INNOVATION, The innovator ex- 
pands his description of the innovation by referring to the Task 1 proce- 
dures. The assessor-consultant may participate in this phase, however, 
this is not a necessity. A sample Descriptive Data Sheet prepared by the 
innovator might appear as follows. Task and subtask references are identi- 
fied where appropridte. 

EX?IIBIT: DESCRIPTIVE DATA SHEET 



NAME OF THE INNOVATION ^ 
3-D Rfoceduraf^frainer 

OBJECTIVES AND SUMMARY DESCRIPTION (01.01) 

The objective is to increase the efficiency and retentivity of the 
learning of operating procedures on electronic equipment. The device 
consists of the "shell of the electronic gear on which the procedures 
are to be learned. Within this shell, special circuits are hooked up 
to a computer which simulates the behavior of the equipment to the 



Problem 
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EXHIBIT: DESCRIPTIVE DATA SHEET (Continued) .-. . . ' 

actions of the student, but also provides corrective and other in- 
structional guidance. The computer is prograrmiable so as 'to adapt 
to the learning objectives. 

DELIVERY SYSTEM (01.02^ 

.The innov'ation centers .around a delivery system for procedures learn-' 
1ng._ The trainer displaces the operational live equipment that Is used 
for^instructional purposes and the instructional manuals that guide the 
student in his learning the procedure. 

Instructional Process . * O 

The device enables the student to perfom manual procedures with actual 
displays and controls and their relationships. He also gets instruc- 
tional feedback and guidance from an aux'lliary display screen controlled 
by the computer The student actually learns by doing the task, with 
supplemental individualized help and direction controlled by the re- 
sponses he has alreWy made, and should be making next. This should ■ 
resu t in faster learning, better remembering, and higher procedural 
level skill. The device is a form of simulator. 

JOBS AND TASKS (01.03) , 

Of primary relevance are those jobs that deal' with the external displays 
and controls of intact electronic equipment. This may include test 
equipment and operator training in. sequential procedures on the equio- 
jtient such as turn on, start up, checkout, and operation. Also, mainte- 
nance tasks done on the outside" of the intact electronic'equipment' such 
as checkout, alignment, and so forth. ^ r 

Equipment Orientation 

Electronic equipment. No restrictions on equipment typfe. 

Wo rk Function ' * '• 

Sequential procedures: reading indicators; activating controls as pre- 
scribed in sequence; recognizing deviations from "normal" and taking 
specified action. ' ■ . ^ 

Operational Environment ' • 

Any _lab environment: no physical motion; normal temperatures. The 
student is unstressed except as may be artifically induced by verbal 
means Space used by the tY-ainer is practically equivalent to that 
used by operational equipment for training. 

/ 
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EXHIBIT: DESCRIPTIVE DATASHEET (Continued)... 
STUDENT ATTRIBUTES (01.04^' ^ 

May; help slow learners of routine tasks. Student attribute factors 
are not a major target of the innovation. 

Prior Training and Experience 

The. trainer could be an efficient way of providing refresher training, 
but this has not been investigated as a justification.' 

Attitudes and Motivations « 

These should be limproved as a by-product of the student getting im- 
personal, individual, and continuous practice rather than waiting around 
for the instructor s attention, or making mistakes and not knowing what 
he didincorrectly for substantial periods of time. The student may, 
early in learning the task, get a proper sense of pace which can be 
forced by the programming of the instructional guidance 

Aptitudes ' . • 

* • * . 

These may be affected," but only as a bv-oroduct of other factors. The 
trainer can reduce the dependence of instructor and student on the stu- 
dent's- ability to learn symbolic materials which may not be highly cor- 
related with his ability to learn the procedures by actual physical 
operati ons . . ^ 

Social Factors • ' 

No intended influence'. 

• Setbacks and Attrition ' 

The reduction of these is a^econdary target. 

Student Evaluation 

A continuous, objecti</e record of the errors made by the student is main- 
tamed, riiis should be 'superior to instructor evaluations. The device 
may be used to keep «-the s-tudent practicing"unti 1 the desired' performance 
criterion is reached. ^ 

EMPIRICAL DATA (01.05) 

StydyperfOHDed on two pieces of equipment. 



TAEG REPORT NO. 12-3 
EXHIBIT: DESCRIPTIVE DATA SHEET (Continued)../ 

Quantitative Benefits 

— ff 

Measured Results 

Control: Normal training operations. 

Exper: Use of innovation on same equipments, same procedures 
. to be learned. 

Results: Tested on live equipment after equal hours of elapsed 
time. 

1. Immediately after training: Exper were 40^ superior. 

2. Retest one month later: Exp^r 60% superior. (Equivalent 
to reducing OJT from 4 weeks ^o 1 week.) 

3. Test six months after on job: 

Exper: 30% rated "outstanding'"- on job. 
Contr: 5% rated as "outstanding." 

4. Attrition 

Exper: 1% 
Contr: 7% 

CriterloPi Variabiles for Measuring Training Effectiveness 

Validity ^nd completeness in level of proficiency. . 

Target: Higher level ' of operational proficiency in proce- 
dures performance. 

Quantitative Costs 

Estimated operation costs for innovation, 8 years: $7.00 per student 

hour. 

Present operating cost for training: $8.50 per student 

— ^.^^ hour. 

txper: One instructor per 20 students. 
Contr: One instructor per 4 students. * 
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EXHIBIT: DESCRIPTIVE DATA SHEET (Continued) . . . 

Estimates of Magnitudes of Benefits/Savings under Actual Conditions 
80% reduction in attrition rate. 
75% savings in OJT per incumbent. 
Reduction in standby and spare equipment. 
Decreases in mission failures. 



ESTIMATES OF THE PRACTICAL IMPORTANCE OF THE TRAINING PRODUCT (01.06) 



TASK 2 - DEVELOP/EXAMINE ALTERNATIVES TO THE INNOVATION. The development of 
alternatives may require extensive study, reference to the literature, and 
consultation with experts. In this scenario, the assessment should be per- 
formed in two steps. First, alternatives generated should be considered 
against the same objectives as the original proposal (3-D Trainer). Second, 
any additional benefits specific to the alternative should be Identified and 
assessed. The following briefly describes an alternative to the 3-D Trainer. 
If this should become an important consideration in the decision process, it 
will be subjected to the same descriptive procedure (Task 1) and analysis as 
the original proposal.- - 

EXHIBIT: INNOVATION ALTERNATIVES 

■• I 

CMI Display Terminals 
^ Micrbfiche f rames random selection by computer. 
Get high rTBuiated-perfonnance proficiency. 
Tjfien final training on live equipment. 
Major yAdvantaqe 



Practital procedural proficiency: crucial importance to operations. 



RESOURCE REQUIREMENTS FOR FURTHER STUDY (01.07) 
Technical development: $12 million over 3 years. 
Empirical studies: $ 3 million over 3 years. 




All teChnolo 



gy in place. 



Attrition reduction from 7% to 5%. 



Student/instructor rcitio increased from 4 to 5. 
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EXHIBIT: INNOVATION ALTERNATIVES (Continued) .. . 
Major Liability 

Slight cost Increase ($4,375.00 to $5,000.00). 

wfre se?ected ZiZT^' f^O^LE. The following risk statement. 

r^[pnn^ L t!'^°'" '^n^ ^^^^^^^^9 in eacH of the major potential risk 
categories. They are applicable to the 3-D Procedural Trainer. 

EXHIBIT: PRELIMINARY FEASIBILITY PROFILE .(Statements of Risk) 

IMPORTANCE TO NAVY MISSION r03.on 

Of moderate importance.' 

ORGANIZATIONAL COMPATIBILITY (03.02) 

Irrelevant to Navy command and career structures. 

GOALS/POLICY COMPATIBIL ITY (03.0:^V 

Support of policy to Increase practical skill levels. 

STATE OF THE ART (03.04) .. . 

•Moderate probability of success with some continued R&D effort! 
Problems: 

1. Hardware linkages of equipment to computer sensors. 
, 2. Simulation of temporal response to operation actions. 

3. Reliability of hardware/software performance. 

4. Lead time needed to- program the training functions on device. 
FUNDING (03.05) 

iS % large investment to bring the Innovation to an operational 
level of implementation (near maximum that can be conmltted to one 
project). 

TECHNICAL SUPPORT (03.06^ 

Some difficlilty In getting R&D talent and implementation talent. 
ATTITUDINAL ACCEPTANCE (03.07) 

Moderate but temporary resistance by instructional conmunity, especially 

svst mlv^ b '1 equipment reliability! or 

system avaiiablnty and responsiveness. 
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EXHIBIT: INNOVATION ALTERNATIVES (Continued)... 



Student acceptance. 

PRELIMINARY FEASIBILITY PROFILE SUMMARY (03.08) 

Figure V-1 summarizes the analysis from 03.01 through 03.07, showing 
the innovation as moderately risky with a number of risk reduction 
projects and associated dollars requi red to bring the implementation 
and use of the innovation to an acceptable level of risk. 

o 

Task 4 - PERFORM' ANALYTIC FEASIBILITY ASSESSMENT. The preliminary assess- 
ment of risks performed in Task 3 is refined at this point; success pro- 
babilities are determined, projects are costed, and a decision variahle 
IS calculated based upon the tangible factors. The following are examples 
of results from Task 4 procedures applied to the analysis of the 3-D 
Procedural Trainer. 

EXHIBIT: ANALYTIC FEASIBILITY ASSESSMENT 
POTENTIALLY HIGH RISK AREAS (04.01) 
State of the Art (SA) 

SAl. Training laboratory locations will present some problems 
in tying student stations to a remote computer facility. 
Present technology permits line lengths of up to 2000 
feet to be driven at reasonable cost. Some facilities 
are a half-mile from the computer site. R&D efforts 
should be devoted to finding a technology capable of 
driving lines up to one mile at reasonable cost. 

SA2. PK-esent computational speeds would cause delays of up to 
4 seconds in responding to operation actions. This may 
affect student acceptance of the device, but more so, 
may impair the learning of certain operational procedures 
where a sequence of actions must be learned and where 
more rapid feedback may be essential. An assessment 
(study) of the approximate amount of training taking 
pi ace on actual equipment, which requires faster response 
that is now possible, should be made. Additional study 
should also be directed at determining student attitude 
effects, especially as they might relate to decreased 
learning effectiveness. 

SA3. Availability of student trainers will be the key to achiev- 
ing desired objectives. Present computer reliability would 
appear to indicate an availability of between 90% and '95%. 
It must be determined if sufficient training contingency 
plans can, be adopted to cope with this level of reliability. 

SA4. The proposed trainer is general purpose and'its uniqueness 
to the training situation will primarily result from- 
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EXHIBIT: ANALYTIC FEASIBILITY ASSESSMENT (Continued)... 



programming the interrelationships between operator con- 
trols and feedback signals (lights, meters , etc. ) . Pro- 
gramming time was assessed for the experimental applica- 
tionj but no specific techniques (e.g., high level 
language, subroutine libraries, etc.] were analyzed for 
speeding the preparation of programs. A major study 
should be undertaRen to investigate possible techniques 
. for improving programming efficiency. 

Attitudinal Acceptance (AA)' ' 

AAl.. Acceptance of the instructional community wil 1 , to a 
great extent, be a function of successful ly, overcoming 
the risk items stated above. Excessive system downtime 
or slow response time could seriously impact student 
acceptance. Appropriate orienting presentations 
should be prepared to help overcome rer,istance of 
administrators and students. 

SUCCESS PROBA BILITIES (04.02) 

Implementation Risks 75% (65% to 85%) 

SAl - Hardware line linkages. 

SA3 - Hardware/software reliabilitv. 

SA4 - t'rogramming time. 
User Acceptar ,ce Hickc 50% (40% to 60%) 

SA2 - Response delays. 

SA3 - Hardware/r.oftware reliability. 
. AAl - III. proper user orientation. 
IMPLEMENTATION PROJE CTS (04.03) 
Project Name 

User Orientation for 3-D Trainer (AAl) 

Abstract " . 

Implementation of the 3-D Procedural Trainer throughout the training 
command will be accompanied by a great deal of resistance, especially 
at the command levels. One solution to this problem is to ensure 
that an adequate understanding of the device exists with officers 
and staff who will be responsible to manage the implementation and 
operation of these devices within their command. This will be 
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EXHIBIT: ANALYTIC FEASIBILITY ASSESSICNT (Continued) .. . ' 

uu aeveiop tne movie at an estinwted cost of $54,215. Develooment 
will be scheduled to begin May 1976 and complete April ^177;°^"*"* 

Objectives , 

Develop l5-m1nute color movie on 3-D Procedural Trainer. 

Improve acceptance of users at comnand level. 
Project Plan Milestones 

Prepare final implementation plan. jan 1975 

Obtain funding approval. ^g7g 

Recruit personnel. Apr 1976 

Begin design-consultation trips. 1975 

Complete design. ^^^^ 

Begin photography. ^g^g 

Complete photography. P^^^ ^g^^ 

Obtain prints. . 

Apr 1977 

Distribute prints for use. 1977 
Preliminary Cost Estimate 
(See Figure V-2.) 

NOTE: The project description will be as detailed as 
required for documenting the project need. 

REEVALUATE SUCCESS PRQBABTL ITTF<; fn^ 04) 

With successful accomplishment of the risk reduction ^ro^orfc • 
anticipated that success probabilitiL will^e: P^^^'*'' 

Implementation 853; (80X to 90%). 

User Acceptance 90% (85X to 95X). 

Project success is estimated at 95% fgoi tn inn*\ t* 4.u 
package fails. prob,bnu(.1:n?™^i^„'o°4!oTiin"^e*5^^ 
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PRELIMINARY 
COST ESTIMATE 



f NEW CZES REVISEDCZI 


PRDIPPT P'^OJ^CTNAME IPROJECT NO. DATE 
rnujcul USER ORIENTATION FOR 3-D TRAINER j63941.AA.1 30 JULY 1975 


PERSONNEL < j 




JOB 


MAN MONTHS 


COSf/MAN MONTH 


JOB COST 




MOVIE DESIGN LAYOUT (GS-12) 


6 


1855.00 


11130.00 


PHOTOGRAPHY {PH6) 


4 


1630.00 


6520.00 




ADMINISTRATIVE 


2 


970.00 


1940.00 












$ 19590.00 


EQUIPMENT 






OESCRIPTION 


QUANTITY 


UNIT COST 


COST 




PROTOTYPE DEVICE FOR USE IN 


1 


12400.00 


12400.00 




DESIGN OF SEQUENCES AND 










FILMING 








$12400.00 










SPACE 


S 


REASON 


SQUARE FEET 


CCST/SQ. FT. 


COST 


































TRAVEL (NON-STUDENT) 






FROM - TO - REASON 


NO. OF TRIPS 


cost /TRIP 


COST 


MULTI-LOCATION TRAVEL FOR 


20 


280.00 


5600.00 


CONSULTATION AND FILMING 
















H 










$ abUO.OO 


STUDENTS 


$ 


NO. OF STUDENTS - REASON 


NO. OF DAYS 


COST/STUDENT DAY 


COST 


































OTHER 


$ 16625.00 


DESCRIPTION 


MAN MONTHS/QTY. 


UNIT COST 


COST 


PROCESSING AND REPRODUCTION 


1 


5625,00 


5625.00 


OF 100 COPIES 


100 


110.00 


11000.00 


















1 TOTAL PROJECT COSTS 


£54215.00 



FIGURE V-2. RISK REDUCTION PROJECT COST ESTIMATE 
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EXHIBIT: ANALYTIC FEASIBILITY ASSESSMENT (Continued):.. 

PRELIMINARY RANGE -OF- EFFECT (04.05) 
Preliminary Range "of Coucses 

Based upon the experimental results in course XXXXo a preliminary re 
view of similar type courses shows the following potential for appli 
cation of the 3-D Procedural Trainer. 



Course Course Length Average Annual AOB 

XXXX 15 day's 6500 

YYY 9 days' 1300 

(etc.) ? 



Major Cost Categories 


(Projected Differences) 




Student Salaries 


+$ 480K 


(Savings) 


Student Travel 


+$ 360K 


(Savings) 


Instructor Salary 


+$ 4,375K 


(Savings) 


Equipment 


~$ 537K 


(Cost) 


On-Job-Training 


+$23,364K 


(Savings) 


Net Savings 


$28,042K 





NOTE: The above savings assume no risk reduction 
expenditures and that implementation is 
successful and there is user acceptance, so 
that the full eight years of savings are 
realized. This cost wilKbe assigned to 
Decision Tree Outcome A. (See Figure V-3.) 

Outcome B (Assumptions) 

Two-year cycle for measuring user acceptance. 

Existing pla^n will be implemented for firwil six years of planning 
period. 

• Original equipment will be purchased. 

• 3-D Trainers will be scrapped (no salvage value). 
Anticipated Cost - $3,800K. 
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, EXHIBIT: ANALYTIC FEASIBILITY ASSESSMENT (Continued)... ^ 

NOTE: Similar analyses would be performed for 
each outcome. 

The savings and costs, as well as the estimated success probabilities, 
are placed on the degision tree diagram and folded back. The prelim- 
inary decision indicated would be to accept the proposal, assuming 
that the risk reduction projects involving continued .study would be 
undertaken, and their costs considered in subsequent analyses. The 
decision tree is shown in Figure V-3. 

TASK 5 - DETERMINE RAIsE-OF-EFFECT. Feasibility has been examined over a 
limi ted' range of potential application. The total range-of-effect for the ^ 
. 3-D Procedural Trainer ts determined through the analytic steps of Task 5. 

•EXHIBIT: RANGE-OF-EFFECT ANALYSIS 

" CHOOSE MAJOR ROUTE FOR DETERMINING RANGE-OF-EFFECT (05.01) 

Potential Displaceablo Cost in' Instructional Vehicles (Route 05.02) 

Instructors. 

Live equipment used for training purposes. 

Procedure trainers. 
Potential Outcome of Training in a Skill (Route 05.09) 
Search st^aitegy: 

Determine relevant courses. 

Derive identification of instructors and instructional devices. 

DEVELOP BACKGROUND FOR TASK CONTENT TRAINING ANALYSIS (05.09) 

IDENTIFY THE LEARNING OBJECTIVE (05.10) . 

See innovation description: * 

Practical skill training jn routinized procedures applied exter- 
nally to electronic equipment and maintenance. 

MAKE PRELIMINARY EVALUATION OF BENEFIT IMPORTANCE (05.11) 

Target training objective is extremely important. 

Magni tudeS' of potential iillprovement are very substantial. 

»• 

Er|c ^"^^ 
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EXHIBIT: RANGE-5f|-EFFECT ANALYSIS (Continued) .. . 

DETERMINE IF BENEFIT APPLIES TO SKILL OR KNOWLEDGE TRAINING (05.12) 
This is practical skill training, so follow Route 05.13. 
Type of skill: Electronic equipment, procedural. - 
SELECT TASK STRUCTURE ELEMENT OR TASK "FUNCTION" (05.13) 
Include skill practice in procedural context of: 

Goal -image ' ' , 

. ' Scan-detect 
liBentify 
[nterpret 

Manipulation of-equipment controls. 

Do not search on these elements; do search on procedures, operating 
and maintenance, op electronic equipment. 

IDENiriFY RELEVANT JOBS/TASKS: CONTINUED SEARCH SPECIFICATIONS (05.14) 

Job Iciass: Electronic equipment 6perations; maintenance. ' * 

Tasif Characteristics: 

Enijipment: blectronic, having manual procedures associated with 
it. 

t> 

Tools: Test equipment for electronic equipment servicing ' 
and checkout. 

Reference Information: Procedures, manuals of instruction. 



Examples of Task Titles: 
Start-up procedure 

.Checkout procedure ^ 
Operating procedure , ' . 
Alignment procedure, adjustment procedure 
Diagnostic or troubleshooting -procedure, external . 
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EXHIBIT: RANGE-OF-EFFECT ANALYSIS, (Continued) .. . 

SELECT RELEVANT STAGE OF LEARNING: CONTINUED SEARC H SPECIFICATIONS 
(05.15) \ ^ 

Not orientation or familiarization or theory of operation.* 

Not fiomenclature and identifications.* 

Mediated procedures.- 

Unusual task conditions (partial). 

Automated performance (partial). ; * 

IDENTIFY COURSE RELEVANT TO INNOVATION (05.16) . 

AN/SRN-12 Omega Receiving Set Maintenance course. 

Total training time: 2 weeks. 

Practical training (lab work): 6 days. 

IDENTIFY RELEVANT TRAINING UNIT(S) WITHIN COURSE (05.17) 

AN/SRN-12 Omega Receiving Set Maintenance course. 

Relevant objectives in training: 

Receiving set operation procedures. 

Lane count determination procedures. 

Preventive maintenance procedures using test equipment. 

' N£t relevant: troubleshooting inside covers of equipment. 

.£siiiiiatedJ:)caii^^ on equipmentr 6 days, 

Estimated training time for relevant practical work: 3 days. 

ASSESS THE BENEFIT PATTERN (05.18) 

AN/SRN-12 Omega Receiving Set Maintenance course. 

Relevant training target time: 3 days. 

Relevant equipment: AN/SRN-12 Omega Receiving. Set 



*Not cost-effective on this device. 
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EXHIBIT: RANGE -OF-EFFECT ANALYSIS (Continued)... 



Training Benefits 

a. Estimated reduction in attrition rate: 80% reduction in 
rate (from 1% to U). The dollars saved by the attrition ' 
.reduction have been assigned an importance rating of 10. 

' due to present funding pressure. 

r Confidence range — best case, 85% reduction — worst case, 
50% reduction. 

b. Liability to reduction in attrition rate: Inappropriate 
emphasis on evaluating student oh "theory of operation" 
content in course. Would -not affect attrition rate on 
targeted objective, but on the couV;se itself. 

NOTE: If the reduction in attrition 'had value 
other tfian in the dollars saved, then it 
would be assigned an importance level as 
other nonquantifiable variables are. This 
might be the case where the particular 
manpower pool is in short supply. 

c. Reduction in aptitude requirements: Potential, but not 
quantified. 

d. Increased flexibility in training program development. 

[ 

Importance (rating of 2). / • . 
ON-THE-JOB BENEFITS (05.25) ' ' - ^ / 

a. Estimate of normal OJT before acceptable performance: 4 weeks. 

Estimate of OJT with innovation: 1 week! ' 

VThe dollars saved by the OJT reduction^, as with attrition 
'dollars, have been 'assigned an importance rating of lo''. 

Confidence range best case, .5^Week with innovation — 
worst case, 1.5 weeks. * * 

NOTE: As with attrition, if increased availability 
of time had a value .independent of dollars, 
it would be treated as a separate variable. 

b. Reduction in units of standby equipment? Estimated 30% 'reduction 

Confidence range best case,, 50% reduction -- worst case, 
10% reduction. 

Importance (rating of 1). ^ ' ^ 

V.-19 * ; 330 i 
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TASK 6 -"PERFORM COST-BENEFITS ANALYSIS. The analysis performed in Task 4 
will be refined to use more precise cost data, to assess these costs over 
a broader and more exacting range-of-effect, and to include In the analy- 
sis some of the less tangible benefits previously Identified. 

EXHIBIT: COST-BENEFITS ANALYSIS 

ASSEMBLE DATA (06.01) 

Obtain Items From Prior Tasks 

Decision tree from completion of Task 4. 

List of courses and jobs from Task 5 to which Innovation Is expected 
to apply. ^ 

Benefits pattel-n to be applied to courses and jobs. Including magni- 
tude change (amount of training; I.e., course length and number of 
graduates to which the Innovation Is considered applicable,- etc.) and 
confidence limits. 

Project Descriptive Data and Project File from Task^, 4, and 5, 



Derive Expected Values of Variables 



The expected values for the cost models and decision tree Input vari- 
ables should be calculated and tabulated in a form as follows: 

^^^^ ' ° Standard 

Relevant Variable Importance Likely Pessimistic Optimistic Expected Deviation 



Attrition 

and 1 0 

OJT Time Dollars 

Trng. Prog. Dev. 2 
Standby Equip. 1 



User Acceptance 
(w/o' projects) 

User Acceptance 
(with projects) 

Implementation 
Success (w/o projects) 

Implementation 
Success (with projects) 

Project Success 



ERIC 



.014 

1 .0 wk 

Increase 
Flexib. 

Reduced 
30% 

.50 
.90 
.75 
.85 
.95 
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.035 
1.5 wk 



.01 

.5 wk 



.017 
1 wk 



.004 
.167 



.40 
.85 

.65^ 

.70 

.90 



.60 
.95 
.85 
.80 

1.00 



.50 
.90 
.75 
.85 
.95 



.033 
.017 
.033 
.017 
.017 
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EXHIBIT: COST-BENEFITS ANALYSIS (Continued)... 

PREPARE COST MOPa CASES AND INPUT DATA (05.02) 

^ ... . •■ 

Develop Qutcoirie Scenarios " 

A - Attrition will be reduced from .07 to .01. ^ 

Instr./Stud. Position Ratio will go from '.25 to ,05. 

Equipment Life will go from 5 years to 10 years. 

Equipment Unit Cost will qo from $4375 to $5625. • ' 

OJT time would be reduced from. 4 weeks to 1 week. 

All other cost data from the formerly planned program to the 
proposed program will remain the same. 

B - Attrition .could fail to reach .01 and will stay around .04. 

Two years will be required to determine acceptance. 

Recovery will be accomplished by reinsti tutina the baseline or 
existing plan on which the improvements in Outcome A were based. 

Trainers will have no salvage value at end of two-year fieri od. 
No OJT time will be saved. ' ' ' 

C - Approximately $700K in development costs will be incurred. 

Approximately $300K will be required to expedite equipment 
procurement to revert to -existing program plan. 

No OJT time will be saved. 

0 - Neutral Outcome. 

NOTE: These outcome scenarios are highly simpli- 
fied and in actual practice would"^ require 
considerably. more analysis. 

Prepare Cost Model Input Data Sheets 

• A run of the existing program plan is made by preparing input 
data as shown in Figure V-4 and V--5 for the Training Cost Model. 

• A run of the proposed program is made by preparing input data 
sheets as shown in Figure V-6 and V-7 for the Training Cost 
Model. ^ . 
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EXHIBIT: COST-BENEFITS ANALYSIS (Continued) . . . 



• Input to the Job Cost Model will be as follows: 

3 weeks OJT time saved. 
Annual Salary Cost is $11,441. 

Annual numbers of graduates used in the input to the Train- 
ing Cost Model will be the basis for job incumbents. 

• To derive costs for Outcome B, input data are prepared for a ■ 
two-year run with data for the proposed program, and a six- 
year run with data for the existing program plan. Examples 

. of input data sheets to accomplish this are shown in Figures 
V-8 through V-11.. 

NOTE: The financial analysis of Task 7 will use 
costs arid savings separated by R&D, Invest- 
ment, or Recurring. To save possible re- 
classification later, this categorization 
of costs should be considered at this time. 

Ensure True "Opportunity Costs" Have Been Used 

All costs used are true "opportunity costs." 

The basis for savings in job incumbent's time was straight salary, 
exclusive of retirement benefits, etc. 

Input Factors 

Time saved per incumbent: 3 weeks/year. 
Number of incumbent equals annual graduates. 
Salary cost: $11,141. 
RUN APPROPRIATE COST MODELS (06.03) 
Training Cost Model Outputs 

The Training Cost Model cost output data for the existing and proposed 
programs are shown in Figure V-12 and V-13. The Discounted Constant 
Dollar value for each program is shown below, and the incremental value 
calculated: 



OUTCOME A 



ERIC 



Existing 
Proposed 

Outcome A Incr. Training 

V-27 



$ 57,361,408 
47,778,656 



Cost 
Cost 



$ 

^37 



9,582,752 Savings 
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EXHIBIT: COST-BENEFITS ANALYSIS (Continued) .. . 

The cost output data for Outcome B'are shown in Figure V-14 and V-15 
and are .totaled below. 

OUTCOME B 

Proposed (2 years) ' $ 16.,614.874 Cost 

Existing (6 years)- 38,992,672 Cost 

Outcome B Total Training $ 55,607,546 Cost 

The incremental value calculated .between the existing program and the 
proposed program, considering Outcome B, is: , « y^'*'" 

n^'i'^'^^n T .1 ^ 57,361 ,408 Cost 

Outcome B Total. ^ 55,607.546 Cost 

Outcome BIncr. Training $ 1,753,862 Savings 

OUTCO^E C • ' ' 

"rft^ $ 700.000 Cost 

Expedite Costs 300.000 Cost 

Outcome CIncr. Training . $ 1,000,000 Cost 

Job Cost Model Outputs 

Only the results are shown, since no Job .Cost Model was available." 
OUTCOME A 

4904 man-years and $37,086,,976 savings. 
All other outcomes are zero. , ; 

DETERMINE BENEFlf-^LIABILITY VALUES (06.04) 
Use Va,riable Data Assembled in 06.01 

Immi Importance 
, Standby paVts and equipment reduced in Fleet 1 
.Flexibility in program development z 
Rank Outcome for Each Variable 

The outcomes are ranked as shown in the first three columns of Figure 
consJde^ed'neJSraL' ''''''' 
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RANK 



OUTCOME 
LETTER 



D 



OUTCOME NAME 



ACCEPT, IMPLEMENT, IMPROVE 



REJECT 



ACCEPT, FAIL TO IMPLEMENT 



ACCEPT, IMPLEMENT, NO IMPROVEMENT 



r.t'P.'rill^ BENEFIT RANKING AND EVALUATION: 

STANDBY PARTS AND EQUIPMENT REDUCED IN FLEET 
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ERIC 



RANK 



OUTCOME 
LITTER 



OUTCOME NAME 



ACCEPT, IMPLEMENT, IMPROVE 



REJECT 



VALUE SCORE 



ACCEPT, IMPLEMENT, NO IMPROVEMENT 



ACCEPT, FAIL TO IMPLEMENT 



FIGURE V-17. BENEFIT RANKING AND EVALUATION- 
FLEXIBILITY IN TRAINING PROGRAM DEVELOPMENT 
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EXHIBIT: COST-BENEFITS ANALYSIS (Continued)... 
Scale the Ranked Outcomes 

The first and last ranked outcomes become 100 and 0, respectively, 
as shown in Figures V-16 and V-17. 

Benefit: Standby parts and equipment reduced in Fleet 
A = 100. 

B = 0 . . 

D is neutral 

A is four times more favorable than B is unfavorable, therefore, 
the scale from A to D is four times the distance D to B. 

■ D = 20 

C is only half as unfavorable as B 
C = 10 

(See Figure V-16. ) 

Benefit: Flexibility in training program development 

This follows the same logic as for the previous benefit analyzed. 
(See Figure V-17 for results.) 

REFINE SUCCESS PROBABILITIES (06.05 - Use 04.02 and 04.04) 

The reanalysis of the success probabilities shows no change from those 
estimated in Task 4 and assembled as input to Task 6 in 06.01. 

User Acceptance (w/o projects) 50% (40% to 60%) 

User Acceptance (with projects) 90% (85% ot 95%) 

Implementation Success (w/o projects) 75% (65% to 85%) 

TiTD^ementation Success (with projects) 85% (70% to 80%) 

Project Success 95% (90% to 100%) 

If projects fail, use user acceptance and implementation success w/o 
projects factors. 
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EXHIBIT: COST-BENEFITS ANALYSIS (Continued)... 



DEVELOP DECISION TREE ASSESSMENT (06.06) 
Data Requirements 

Cost model incremental outputs from 06.03. 

Benefits ranking and valuation data from 06.04. 

Refined success probabilities from 06.05. 

Develop Equi valent ^Dollars for Nonquantified Benefits 

Standby equipment variable. 

The importance ratio of 1:10 causes this benefit scale of 0-100 
to be adjusted as shown in Figure V-18. 

Flexibility in Training Program Development 

(See results on Figure V-18.) 

NOTE: The sum of the ATTRITION and OJT TIME 
benefits measured in actual dollars, 
was used as the basis for scaling the 
two nonquantified variables. 

Display Range of Values for Each Outcome 

Figure V-19 shows the relationship between the valuations for each 
benefit, adjusted for importance. The total value (in equivalent 
dollars) for each outcome is summarized at the bottom of the chart. 
These values are entered directly onto the decision tree for Outcomes 
A through D. 

r ■ 

Complete Decision Tree Assessment Diagram 

The values from Figure V-19 for each outcome are entered on to. the de- 
cision trfee, as are the values for the readjusted success probabilities 
from 06.05. Figure V-20 shows a completed decision tree diagram. 

Foldback the Decision Tree 

The folded back decision tree of Figure V-20 confirms more strongly, 
the indicated decision tree path established ir, Task 4, using the 
preliminary range-of-effect data. The calculated decision variable 
is 32.5, favoring acceptance and undertaking of the risk reduction 
projects. 
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EXHIBIT: COST-BENEFITS ANALYSIS (Continued) .. . 

PERFORM SENSITTVT TY ANALYSIS (Of,. 07) 
Review Sens<t. i.vltv Profile for Each Mndol 

Variable 



OJT Time 
Attrition 



Confidence Range 
1.5 to .5 weeks 
3.5% to ]% 



ERIC 



User Acceptance (w/o projects) 40% to 60% ' 

(An output report as' shown in the 06.07 procedures 
was not generated, since none of the variables cau.PH 
cen'ar 0 I'.l ^^^^ ^^'^'dlrlc range 

calculS?[lc h^ '"^""^^d *° sensitivity 
^equi™ r °' ''''''''' "^""^1 computational 

EXHIBIT: FINANCIAL ANALYSIS 
ASSEMBLE DATA rn7.m) • ^ . 

Cost Model Output Data From 06.03 
Risk reduction project costs. 
Training savings. 
OvlT time savings. 

Alternative Approach tn M^Pt^n^ .n^. ^- . ^ti ve From Task 7 
' DEVELOP COST/ SAVINGS ANALYST^; (07.02) 
Make Required Cost Model Runs 

sstl^'e''S?jfS^•nTSs^^ "??j:^eV? trtZ'\r.ri °^ 

and outputs for this run. ^'^°'^9h V-23 show the inputs 
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EXHIBIT: FINANCIAL ANALYSIS (Continued)..- 



Generate Cost/Savings Reports 

Each alternative is ^assessed for incremental effects against each 
oth^r alternative, and up to three fonnats of Cost/Savings Report 
are generated for each case. 

Cost/Savings Reports for the existing vs. the proposed system have 
been generated in two formats as shown in Figure V-24 and V-25. 
Similar reports in all formats would normally be generated for the 
existing vs.. the alternative system and for the proposed vs. the 
alternative system. ^3 

DEVELOP ALTERNATIVES COMPARISON ANALYSIS (07.03) 
Select Alternatives for Further Consideration 

Each of the two alternatives (the proposed 3-D Procedural Trainer and 
the alternative from Task 2) has been selected for comparative finan- 
cial analysis. 

Organize Data 

Since 'the data hSve been divided previously into R&D, Investment, and 
Recurring, processing can take place diVectly. The R&D costs, for 
economic analysis purposes, are treated as an investment, as are 
equipment purchases. 

Generate Alternatives Analysis Report 

Figure V-26 summarizes the data comparing the proposed 3-D Procedural 
Trainer with the existing (or presently planned) system, the alterna- 
tive approach from Task 2 with the existing system, and the 3-D Trainer 
with the alternative. 

Review Alternatives 

The results portrayed in the Alternatives Analysis Report may be diff- 
icult to interpret because of the values used in this scenario. They 
can be summarized as follows: 

0 The proposed innovatijcn provides about a.75% return on invest- 
ment which is extremely high, and in the absence of excessive . 
risks (already eliminated with the risk reduction projects), 
should be acceptea. 

© The alternative proposal, however, is even more lucrative s*ince 
it requires a negative investment to produce savings. This is 
due to a lesser equipment cost and the fact that no R&D funds" 
were identified to be expended. Thus, it would be favored for 
acceptance over the proposed 3-D Procedural Trainer. 
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EXHIBIT: FINANCIAL ANALYSIS (Continued)... 



• From the foregoing, it is seen that there is a negative incremen- 
tal return on investment between the proposed 3-D Trainer and the 
alternative identified in Task 2. 

• Since the alternative was not subjected to the analyses from Tasks 

4, 5, and 6, these should be performed to determine if it should 
be- accepted without the inves.tment of risk reduction project funds 
If substantial risk reduction project funds are identified, the 
financial analysis may have to be redone. 

PERFORM SENSITIVITY ANALYSIS (07.04) 

Review Sensitivity Profile for Each Model 

This is similar to the analysis in 06.07, however, it would be restricted 
tothe tangible factors measurable in real dollars. The decision variabl 
being tested in this analysis is the return on investment factor. 

Identify Sensitive Variables to be Analyzed 

^^'^"'^^^^ Confidence Range 

Time 1.5 to .5 weeks 

Attrition 3,5^ to 1^ 

(An output report as shown in 06.07 procedures was not 

generated since none of the variables caused a change 

in' the decision across the confidence range.) • • 

TASK 8 - MAKE ACCEPT/REJECT/STUDY DECISION. The major outputs of this task 
will be left to the reader's interpretation, using the guidance dut'lined in 
Task 8. The presentation content, however, will include: 

1. A statement of the problem. 

2. Background data. ' 

3. The objective. 

4. The proposed approach for meeting the objective. 

5. Supporting rationale. 
• Assumptions made. 

® Risks (including projects to bring them under control). 
0 Benefits/liabilities (including savings and costs), c 
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• Results of decision tree analysis. 

§ Results of financial analysis. 

t Plan for accomplishment. 

6. Summary conclusions and recommendations. .\ ^ 

SCENARIJl-N UMBER TWO - COST ANALYSIS AND MEASUREMENTS SYSTEM 

This scenario deals with a subset of the analytic and decision process and 
points out some the considerations in the treatment of variable estimates, 
costs, risk, financial return, and meisurement statistics used by training 
management. 

FINANCIAL ANALYSIS. An identified segment of training presently handles 10,000 
Student Days (DE) per year, at a total cost of $1 million. The cost per SD 
is $100. The $1 million consists of $100,000 in fixed costs and $900,000 in 
variable costs. The total cost of training can, therefore, be expressed: 

TOTAL COST - $100,000 + $90 X STUDENT DAYS 

A highly innovative proposal is submitted which permits students 'to be trained 
at twice the existing rate; however, it will require an additional $200,000 
in annual fixed costs. The proposed training system will have 5000 SD's. The 
two training systems can be described as s.uch:- 



A (EXISTING) B (PROPOSED) 



ANNUAL STUDENT DEMAND 


500 


500 


AVE COURSE LENGTH (DAYS) 


20 


19 


NO. OF STUDENT DAYS (SD'S) 


10,000 ' 


5,000 


TOTAL FIXED COSTS (^$) 


$100,000 


$300,000 


TOTAL VARIABLE COSTS ($) 


$900,000 


$450,000 


TOTAL COST/SD {%) 


$ 100 


. $ 150 


VARIABLE COST/SD ($) . 


$ 90 


$ 90 


COST/GRADUATE ($) 


$ 2 ,000 


$ 1,500 



The proposal obviously sounds enticing, therefol^e, the cost analysis should 
provide management with the appropriate decision-ffraWng structure with which 
to judge the proposal. The key questions are "What are the. relevant costs?" 
and "How does the acceptance of the proposal affect the measurements which 
managers at all levels may be viewing?" 

Before address-ing these questions specifically, it might be well to look at an 
economic analysis of the proposal. Assume that the proposal is to be analyzed 
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=rJnn P^r^""? °^ ^^'^^ ^^^^^ Student demand will be maint-ained 

at 500. An incremental analysis of the proposal gives the following results. 



YEAR 


ANNUAL SAVINGS 


- ANNUAL COST = NET 


DISCOUNTED @ 10% 


1 


450,000 


- 200,000 


= 250,000 


238,500 


s 

2 


450,000 


- 200,000 


= 250,000 


216,750 


' 3 . 


450,000 


- 200,000 


= 250,000 


197,000 


4 


450°, 000 


- 200,000 


= 250,000 


179,250 


5 


. 450,000 


-'' 200,000 


= 250,000 


163,000 






NET PRESENT 


VALUE 


994,500 



If the opportunity cost of, capital were considered to be 10%, then one would 
pay nearly $1 million for such an innovation in training. Even in a situation 
where there is some risk that the. annual demand will not be sustained, and it 
mighflook as ?ol?ows''' '° risk potenti al . the analysis 



YEAR 


PRO J 


DEMAND 


RISK 


ADO DEMAND 


SAVINGS - 


COST 


= NET DISCOUNTED @ 10% 


. 1 


500 


X 


.90 = 


450 


405,000 J 


200,000 


= 205,000 


195,570 


2 


500 


X 


.80 = 


400 


360 ,000 - 


200,000 


= 160,000 


138,720 


3 


500 


X 


.70 = 


350 , 


315,000 - 


200,000 


= 115,000 


90,620 


4 


500 


X 


.60 = 


300 


270,000 - ' 


200,000 


= 70,000 


50,190 


5 


500 


X 


.50 = 


250" 


225,000 - 


200,000 


= 25,000 


16,300 












NET PRESENT 


VALUE 




491 ,400 
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AHH^I.-nn ? . .Jl Stated one might pay nearly $500,000 for the innovation 
Additional risk could be associated with the training length. For example, 
assume that the probabilities of achieving, the projected training length reduc- 
tions were as follows. (Remember the existing trainiag length is 20 days while 
the proposed length, with the innovation, is 10 days.) • 

NEW COURSE LENGTH , PROB. OF ACHIEVING EFF REDUCTION 

0 
0 

0^ Eff Course 

0 Length = 

0 10.95 Days 

0 

.70 Savings in 

K 30 Course Length = 

1.80 9.05 Days 
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20 




0 


19 




0 


18 




0 


17 




0 


16 




0 


15 




0 


14 




-.05 


13 




.10 


12 




.15 
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NEW COURSE LENGTH PROB. OF ACHIEVING EFF REDUCTION (Continued)... 

11 .20 2.20 

10 .45 4.50 - 

9 .05' .45 

10.95 

If this dimension of risk .is added to the previous analysis, rather than the 
adjusted demand being multiplied by a 10-day savings per student, it will only 
be multiplied by 9.05. That analysis would give the following results. 



YR ADJ 


DEMAND 


X 


DAYS SAVED 


•- TOTAL 
DAYS 
= SAVED 


COST/ 
X SD 


= SAVINGS - 


COST 


= NET 


DISC ? 

10^^ 


1- 


450 


X 


9.05 


= 4072.5 


X 9a- 


= 366,525 - 


200,000 


= 166,525 


158, 8P5 


2 


400 


X 


9.05 


= 3620^ 


X 90 


= 325,800 - 


200,000 


= 125,800 


109,069 


3 


350 


X 


9.05 


= 3167.5 


X 90 


= 285,275 - 


200 ,000 


= 85,275 


67,197 


4 


300 


X 


9.05 ' 


= 2715 


X 90 


= 244,350 - 


200,000 


= 44,350 


31 ,799 


'5 


250 


X 


9.05 


= 2262.5 


X 90 


= 203,625 - 


200,000 


= 3,625 


2,364 














NET PRESENT 


VALUE 


369,294 



The value of the proposal is down to $370K, however, it still appears attractive. 

From the precedi ng examples , it can be seen that the relevant costs (or costs 
avoided), are: - 



1. The additional fixed cost ($200,000). 

2. Th'e variable costs saved ($90 per Student Day). 

These costs were adjusted in several ways to account for risk, and the' net 
savings were discounted at a TO^ rate to account for^the time value of money. 
An additional adjustment could have been made on the annual fixed cost if it 
were felt that some risk was attached to that value remaining at the speci- 
fied level . 

MEASUREMENTS ANALYSIS. An equivalent concern with adoption of a proposal such 
as this relates to the effects it might have on the values reported to manage- 
ment for measuring and controlling the training system. The use of ratios is 
quite common, such as cost/student hour, cost/student day, cost/graduate, etc. 
The introduction of the proposed innovation will affect certain ratios of 
these types, some favorably, others adversely. For example, the proposed 
innovation will cause a 50% increase in student day costs, while it will cause • 
a 25% decrease in the cost per graduate. The following comparison shows hovi 
these changes result. Variables names have been shortened to facilitate 
mathematical handling. 
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VARIABLE NAfC 
Annual Student Demand 
Ave. Course Length (Days) 
No. of Student Days 
Total Fixe.d Costs 
Total Variable Costs 
Total Cost/SD 
Variable Cost/SD 
Cost/Graduate 
'Total Cost 

VARIABLE RELATIONSHIPS 

CSDT = (CFX + CVRB)/SD 

= (CFX + CSDV*TLENGH*GRAD)/(TLENGH*GRAD) 
CGRD = (CFX + CVRB)/GRAD 
For A (Existing) : • . • 

CSDT^= ($100,000. + $90. * 20 * 500)/(20 * 500) 

=■($100,000 + $900,000)/10,000 

= $1,000,000/10,000 
■\ = $m (COST PER STUDENT DAY) 
CGRD;\= ($100,000. + $900,000)/500 



= $1,000,000/500 



= $2,000 (COST PER GRADUATE) 
For B (Proposed) : 

CSDTb= ($300,000. + $90. * 10 * 500)/(10 * 500) 
= $300,000. + $450,000.)/5000 
• = $750,000./5000 . , 

= $150. (COST PER .STUDENT DAY) 



^ >SYMBOL 
GRAD 
TLENGH 
SD 
CFX 
CVRB 
CSDT 
CSDV 
CGRD 
CTOT 
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CGR[)b= ($300,000. + $450,000. )/500 
= $750,000./500 
= $ 1500 . (COST PER GRADUATE) 



A failure to consider the economic justification as well as measurement im- 
pacts, may lead to future misinterpretations of the value of the innovation, 
once it has been implemented. A management measurement and control system 
which focused only'On Cost per Student Day (CSDT) might be shocked at the 
50% increase and may not give proper credit for adoption of an economically 
sound innovation. . . - 

Another important management consideration is with the sensitivity of various* 
measurements to student loads under exissting and proposed conditions. The 
two previously considered measurements ,^ost per Student Day and Cost per 
Graduate, should be analyzed for their profiles of' variance under the two 
training situations, (A) existing system and (B) proposed system. 

In Figure V-Z?, TotaT^Costs and Cost/SD are plotted for the two alternatives. 
It can be seen that' ynder the proposed system (B) , student day costs are 
higher for any number of student days^ and they rise more rapidly as student 
days decrease. However, a more relevant consideration is the break-even point* 
for Cost per Graduate as shown in Figure V-28. If the number of graduates 
drops from 500 to 222, then the cost of operation under the existing system 
and the proposed system is the same ($500,000); therefore, the Cost/Graduate 
is the same ($2222.). At the break-even point, however, the Cost/SD under 
the proposed system will be twice that of the existing system ($225. vs. $112.5) 
This is not a relevant concern. What is of concern is the higher operating 
cost below the break-even point of 222 graduates under the proposed system. 
Thi-s— is due to the higher leverage provided by the increased level costs. A 
higher Cost/Graduate exists, also, under the proposed system, as the number of 
graduates, falls below the break-even point. 

There are several, alternatives for maintaining the continuity of the previous 
measurements if the proposal is accepted- One is to use the Cost/SD (B) as a 
control value for all subsequent numbers of graduates. Even though it is 
higher, future cost problems can be detected from a higher Cost/SD than pro- 
jected at the time of the analysis. Another alternati ve j's to adjust all 
future student day costs by a ratio of the before-to-after costs at the time 
of the analysis. Since this value will- vary depending on the number of 
graduates, it must be ca^l^culated for each level. The procedure would be as 
follows: . 

) ■ . 

Let CSDTq be the Cost/SD measured and observed at some future date. 

Let CSDTf^ be the Cost/SD adjusted to the preproposal base and the value 
that will be presented to management. 

Then: 

CSDTm = CSDTq X CSDT/^ 




\ GRADUATES 



FIGURE V-28. TOTAL COST/COST PER GRADUATE BRJAIC-EVEN ANALYSIS 
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Since CSDT;\ and CSDTb are calculable as a function of numbers of graduates (GRAD) 
at time of proposal analysis, the equation reduces to; 

CSDT,. = CSDTn X 1000 + 18 GRA D 
bOOO + 18 GRAU 

For example, assume 
CSDTq = $160 

GRAD - 500 ■ ■ * 

Then : 

CSDTj^ = $160 .X 1000 + 9000 
600U + 9000 

= $160 X .67 

' , = $107. 

V^^^^il lJ^l^K°.tr.^^^ graduates prior to the proposal, management would have 
ITri^^t ^° CS'^T. = $107,'reflects a cost problem 

which should be investigated further. 

SUMMARY. The purpose of this -scenario has been to unfold several of the more 
important factors 1n evaluating an innovation proposal; namely, the identifica- 
tion of relevant variables, the economic justification process, and the effects 
upon the measurements and control system. 

SCENARIO NUMBER THREE - SHIPBOARD TRAINING WITH MICROFICHE 

The following is an example of the use of ETAM in assessing the potential of 
using microfiche as a medium with a simple M/F viewer for shipboard training 
of courses or course content now taught on shore-based installations. 

In this scenario, it is understood that the assessment opVations are to be 
applied principally to Tasks 1. 3. and 5 in ETAM.. Cost data are not to be 
??fiJ S^^P^^^d* ^""^ general classes of cost information should be iden- 
plnnf major point in this exercise, however, is the use of Task 5, 

Range-of-Effect, structure on this problem. ia:>^ ^, 

m?r.nJiVhI'/''V^-'-^'' '"'^^'^ ^^''^ ' background in the useof 

m_icrofiche for training purposes. He was given several study reports into M/F 
u^. some briefing by Dr. Braby of TAEG. and some literature on extenslSns of 
microimage technology into training, such as the Lincoln Lab technology This 
was the analyst's background to the present assessment results. 

Since an assessment must have, at least as a starting point, a fixed reference 
as to the entity being assessed, it was decided to use the standard microfiche 
and the simple, portable viewer as the "innovation." If instructional or other 
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functions were to be added to this "system" j(such '^s the elaborate Lincoln sys- 
tem contains), these additions would and could be superimposed on the present 
pattern of benefits and costs as modifications or e-xtensions. 

Because this report i s Jji tended 'to have 'instructional value on ETAM as well as 
being a trial operation, the text is fairly extended. Many of the reasons for 
making the stipulations in the assessment that appear here have been reported, 
perhaps at more than- necessary length. Issues have been raised and explored 
which, in the present problem, have led into a dead end-. It is believed that a 
full exposition of this kind enables others, as contributors to the assessment, 
to work, wi th ~,the rationale for a conclusion, as well as with the conclusion i^t- 
self. Thus, it would be surprising if there were not some strong disagreements 
with some of the -opinions asserted as "assessments." Clarification can be pro- 
gressive, however, when the participants to a controversy can sort but whether 
their disagreement is based on assumptions, stTp'ulQtions , rationales, or infer- 
ences and judgments. , • ' 

The preparation of this entire assessment,' including more than 27 pages of text, 
has occupied between four to five days of time. The setting down of the actual 
assessment data in the report -- without the rationales, etc. with the' infor- 
mation at hand, would have taken, between one and two days,^ and closer to one 
than to two. These es,^imates are intended to suggest that it does not neces- 
sarily take forever to go through the range-of-effect structure. As a matter 
of fact, describing the innovation in Task 1 would have taken more than half of- 
the time given to the assessment operayjan through Task 5. - 

EXHIBIT: ETAM EVALUATION OF MICROIMAGE TECHNOLOGY* FOR TRAINING 
TASK 1 - FORMALIZE THE DESCRIPTION OF THE INNOVATION 
MICROFICHE TECHNOLOGY 

Objectives and Summary (01.01) 

The general proposal is aimed at the substitution of a system for 
producing and using micro-images, such as microfiche, for papei^ as 
a medium for individualized instruction on shipboard. The microfiche 
type of card can be coupled with a viewing mechanism that enables 
student responses to be evaluated. The result of the evaluation 
can be the basis for the. selection of what is next presented to th-e 
student. It is also feasible to develop techniques which simplify ■ 
the creation and modification of instructional content by authors, 
using a canputer to assist in formatting, indexing, sequencing, and 
retrieving instructional material, 'the general benefits made possible 
by this technology are: 

1. Reduction of storage space for equivalent content on paper, 
from 50 to 1 , with a potential of 200 to 1. About., the same, 
proportional reduction in weight for transport and distribution 
purposes. 

2. Somewhat greater flexibility in the amount and range of content, 
and in content sequencing: this enables options in sequencing 
that increase instructional capability. 
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EXHIBIT: ETAM EVALUATION OF MICROIMAGE TECHNOLOGY (Continued)... 

3. Substantial reductions in costs of updating and reproducing of " 
images as compared with printing" on paper. These reductions in 
cost can be increased by' computerizing the means whereby sets " 
of images are created; layouts are generated by the author: and 
indexing, retrieval, and collation of images is managed. 

■4. 'The technical feasibility for shifting a large- burden of instruc- 
tion from shore schools to individualized shipboard instruction ' 
in a substantial range of knowledge and in some skills. 

Because of the substantial development and operating costs of the 
, proposed technology, the payoff must come from a large scale of 
application. Furthermore, any implementation should enable exten- 
sions of other technologies, such as update by satellite communica- 
tion, or conversational interaction between the student at a micro- 
image terminal with a central computer,. to be readily integrated to 
. existing facilities. / ^ 

Delivery System and Traini ng Vehicle (01.02) 

Microimages are contained on transparent cards. One of the standard 
sizes IS 4 X 6 . Between 60 to several hundred images are on a card 
Greater reductions are possible. The card is inserted into an optical 
viewer which enlarges the. image to readable size. Substantial advances 
have been made in the quality of viewers, and in compactness of the de- 
vice. The desired image is located manually; movement from one image 
to the next is manual. There are various forms of indexing assistance 
tor locating an image and mechanical devices for advancing image frames 
Manual advance seems, until now, as effective as mechanical aids The 
cost of automatic sequencers would greatly increase the cost of the 
device and increase failure rate and maintenance costs. 

The proposal does not include a firm specification for the properties 
of the viewing device. A reference base, nowever, for further cost- 
benefit analysis and comparison is the simple, portable viewer which 
P^^^n*^*'^^ ^^"■'^^■■'s"* an 8-1/2" x 11" display with a resolution 
of ISO lines to the Inch. The microfiche card is manually Inserted 
.and Images are manually selected and advanced. 

At present, it is assumed that the formal testing of what the student 
learns at tnelsvel of specific course objectives would be made on 
another medium and device, perhaps served by a human instructor. 

Jobs and Tasks Applicable (01.03) . . ' 

Preliminary examination suggests that anything that can be learned 
from print on paper can be learned by microimage. This applies to - 
the large body of knowledge material called "Knov/ledge Factors" in 
occupational standards. To the extent that a knowledge factor is 
necessary to the performance of a skill (or "Practical Factor"), this 
medium is applicable. Most procedural tasks require a knowledge substrate 
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EXHIBIT: ETAM EVALUATION OF MICROIMAGE TECHNOLOGY (Continued)... 



On. this premise, practically all jobs. and job advancement opportunities 
would be candidates for this facility. 

Relevant Student Attributes (01.04) 

The requirement is a reading ability at the level of ;prograitimed in- 
struction fO|r the given instructional topic and trairting objective. 

The student may require smewhat higher levels of seljf-discip^ine 
than would be needed in formal school where time is structured and 
study activities are more or less supervised. Another complication 
could be the fact that on shipboard, the student might feel his pri- 
mary responsibility was his job and properly so but this moti- 
vation could lead to rationalizations that the effort required for 
self-instruction could be postponed. For many students, some exter- 
nally apali^d incentive might need to be applied on a daily or weekly 
basis . / , 

The lack of the stress of schedules whereby schools ^must operate 
classes, however, would enable slower students to spend more time 
in mastering subject matter. The consequence could be a lower wash-^ 
out Vate, at least on the basis of applied ability level. . ) . 

Empirical Data on Benefits and Savings (01 .05) 

Samples of data, both on the potential liabilities and benefits in * 
learning with the microfiche system and its costs, are in process of 
being collected. 

At least indirect evidence shows that microfiche is at least as good 
as fjaper manuals, texts, and programmed instruction. 

The 'physical density of storage of content on microfiche makes it 
fea$ible to make a large variety of instructional material available 
to the learner. This enables him to satisfy his curiosity, and may 
enable additional learning through redundancy: the same idea or 
pattern of ideas may become not only better learned and understood 
(paraphrased in various contexts) but better remembered and potentially 
better applied to appropriate occasions, 

r 
I 

The negative possibility is that most students will tend to learn 
material only to a minimum level of grasp if they are left to set 
their own criterion. This means rapid forgetting, even if the test - 
is passed. An external means of controlling amount of practice and 
criterion level reduces this liability. However, this liability 
exists in printed materials. 

Practical Importance of the Training Product (01.06) 

A large majority of skills has a substantial knowledge component, 
either as background for acquiring the skill, or as operational and 
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EXHIBIT: ETAf'I EVALUATION OF MICROIMAGE TECHNOLOGY (Continued)... 

reference information for exercising the skill. To the extent that 
opportunity and incentive can oe.ereated on shipboard for acquiring 
these knowledges, a double savings can be effected. One is. the re- 
duction in classes and class time in shore installations. A second 
IS the reduction in the proportion of enlistment time during which 
the incumbent is not in oporcvtional status — at least following his 
first enlistment's period of schooling. 

If this innovation can be realized, the achievement of its goals have 
substantial practical impOf^tance, , 

Resource Requirements for Fnrthf^rJUjd^ndJin plementation (01.07) 
A program has not yet been prapareci. 

Although the right hardv^-ere may not be available, no break-through in 
technology is required for microfiche. The an ci ci pated use of holo- 
graphies which could extend th^ instructional possibilities perhaps 
substantially would require some technological break-throughs both " 
in cost and in miniaturization of components. " 

It appears that consideriiD'y more effort may need to be spent in 
determining ways of ubing th^ mediiim and its possibilities for instruc- 
tion. Some useful principles have been invented and applied, but with 
limited empirical data. ^^i- . ■ . 

Although the unit cost of che microfiche viewer is relatively small, 
It is not trivial. But the devalcp!)ient of a coordinated system for' 
generating and adminis c.-iring larye sc?.le curricula would require a 
multi -million dollar effo.'t. 

A somewhat higher level of average talent is likely to be needed to 
develop effective microfiche instructional content than is character- 
It^^i 2^ classroom inatructcrs snd of proqrammed instruction designers 
The latter have the instruction«l machine as a surrogate human instruc- 
tor to. keep the student going through to the end of an instructional- 
segment. The student with book or microfiche. can more readily turn 
off without a sense of ncnccmpleti on of a responsibility. 

TASK 2 - DEVELOP/EXAfvlINE ALTEi^ATIVES TO THE INNOVATION. Instructional material 
on paper is the primary alternative. The microin'sq.? may be cost-justified over 
paper, quite inaependent of other bsnef-ts, 

There may be ' t©chn1 cal alternatives to microfiche and the simple microfiche 
viewer. It is unlikely that they will be lower cost per frame of displayed 
information, but they may turn out to be lov^er overall cost per unit of learned 
product such as a kncwledge objective or a si-lli objective, and across a wider 
range of subject matter and learning levels. ' 

Those comparisons are not a part of the present evaluation, except insofar as 
this evaluation provides a reference for cc?iiparisons. 
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TASK 3 - PRELIMINARY FEASIBILITY PROFILE 

Importance of the Innovati^on to the Projected Navy Mission (03.01) 

Shipboard training is an alternative to s\vore training, and therefore 
cannot be considered crucial. In the event of an extended military 
emergency and full-scalfe mobilization, the ability to upgrade the capa- / 
bill ties of every person, wherever he may be, and to do so rapidly, 
could be crucial. But this is not the case, nor is it a present 
planning consideration. • • , 

Secondly, the training of knowledges is subordinate to training in 
actual job skills- as such v/here, in most cases, the learning and 
application of the knowledge is likely to be most effective and 
efficient. 

Taking both of these arguments into account, the estimate would have 
to read: 

Shipboard training with microfiche and what can be learned . 
from microfiche technology would have an effect of moderate 
importance. 

But taking into account that relatively few pounds of microimages 
as training aiaterial and as job aids (technical manuals) can -dis- 
place tons of paper containing equivalent information, the picture 
changes rather radically: 

The effect can be substantial and important. 
Organizational Compatabi 11 ty (03.02) 

Shipboard training, as contrasted wi th shore-based, training, is 
i rrelevant to Navy command and career- structure, or su pporti ve, of 
career structure. Since the policies of the Navy are to support the 
• fullest possible growth potential of all of its personnel, shipboard 

training is indeed supportive and relevant. 

Goals/PoUcy Conp atabi 1 i ty W1th1n Training and Personnel Organiza- 
• tions (03.0^ 

To the extent that an organization prefers to maintain complete con- 
trol of both the process and the product it is charged with producing, 
it is understandable that the training institutions would prefer full- 
time students studying- in the training establishment itself. The 
student within this jurisdiction has direct accountability only to 
the training officers and management, and thus -the training ipnage- 
ment has the authority to discharge its responsibility for producing 
trained men. 
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On the other hand, tramincj [uanaqemani zbu perceive its objective as 
achieving training at "^ovvest cost and penalty to the NaVy as an oper- 
ational enterprise. In tnl.:; iatiu/ lights shipboard training is an 
extension of the training encerpriso with a concern for cost as well 
as the belief it of the prouucl, 

♦ 

A realistic assessiii'^nt woulo be tr.ai some minor confli cts of a temp- 
orary nature would arise within the training estaBl-riTiment, not on 
the microfiche progrcu! a-: zach^ but agdinst shipboard training on a 
wide and formalized ^cali:. 

Estimate of Sta ie-;jp.)f-th e-Ar c App't cabie to. the Inn ovation (03.04) 

With respect to the technical effecti vf-^ness of instruction with more 
or less ^existi rig microfiche techno'iogy. 1 L seems realistic to expect 
a hicjh DL^babi li ty j)f^^ l^ast a moderate R&D effort. 

This eT*f6rt would be^ii.oTrt 'p^O:'r::ob'^y sptjnt in che techniques of 
successful instruction vvich vhis i.ouium^ and in techniques of gener- 
ating or organizing such instruction. 

Estimate Furth er RaD Fuh^^^^^^^ t\Q (03.05) 

Programs for R&D in this e/uerj^riso f\(Ve ,'o'c been formulated. But a 
full-scale^.implefiientatlon across u iriae variety of courses^ job skills 
and knowledges, and stuucnc^ v/ccij; require a niorieratc level of funding 
somev^hat above the level of present planned aFiTTntegrated programs. 

It would be possible to i\'t\cm cr<M.n feasibili ty \on ? relatively re- 
stricted scale with re;dt'v^/\v '^'O^T^: fi.a.is. Such a demonstration 
would lack 5 however*, any -w.!, is for essincMt of the economic benefit 
and the instructional ri.ngc and tk.wirrhty of a full-scale develop- 
ment which inciudad con'jj.iut^ri z^^d techniques for preparing and dis- 
seminating training ccnt;-i,. 

Required T echnical Su;^por'. (03.05) 

The kinds of taUht iaai: w^jid be needed for developing these training 
materials, ana for cdnini bearing this c:n;:ire instructional program, 
might be unusual -lUt no;: 'ri\^a. Acleq uate ta ie.it could be made avail- 
able. 

Attitudi nai Accep;r:ance (03.0/) 

In the case of shipboard ^raiiiing, tn^ key requirement for attitudinai 
acceptance will be thut of zhe co^m]iii\d\nc officer of the ship. Very 
reasonably, he feels his primary mif^^ion and responsibility is the 
effective operation of vh^:'::h^p. is unpikely to feel that the 
training of personnel on 'Vi^ >hlp is a legitimate competitor to the 
operational mis:^1on. i: clrioooard inadequacies are traced to this 
"excuse", the coOTjander i^. 'iS'^^ly to downgrade the training mission. 
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This set of values will quickly be transmitted to trainees. Since 
self-instruction requires considerable incentive, the loss of the 
.pHnnc? *^'?j'""u*iu' ^ ^ "^P"'' motivation to take the training 
s{royed opportunity and incentive can be largely de- 

It is unrealistic to expect that commanders brought up in tradi tional 
N^yy doctrine will accept a tradeoff criterion between any threat to 
shipboard effectiveness and training objectives that do not point 
airectly to an increase or at least maintenance of shipboard effective- 
ness The acquisition of. new knowledge and skills for promotion to 
another grade does not point directly to such an increase in the capa- 
Dility of the ongoing mission of the-sh+pr— 

This is a major impediment^ not to microfiche as such, but to the 
instructional environment of 4ts intended use^ 

The assessment must be: The coimiunity in which the innovation is to 
be introduced will respond with sustained hostility . 

Many. potential students on'ship may, even with the encouragement of 
the commander for training, be ambivalent about time and effort spent 
in training that is away from their duties and comrades, or from re- 
laxation. 

TASK 5 - DETERMINE RANGE-OF-EFFECT . Some preliminary coimients are in order 
before plunging into the procedural operations of Task 5. Clearly, one pro- 
posed advantage of microfiche is in its various economies over Its equivalent 
lnuif^''-°^^ Shipboard - plus, perhaps shore-based training and operations. One 
could, without going further Into the specifics of range-of-effect , make dollar 
comparisons of the M/F and paper alternatives^for existing shipboard materia s 
this Jasi 111 one ""''^ °^ '"'"^"^ shipbo&rd paper might be justified on* . 

The second problem is the justification for using M/F for shipboard treininq 
Here, the major question - assuming acceptance of shipboard training by the 
command levels -- is what can be taush^-^n'd learned with the M/F medium? The 

LaL'?^'''^ nnn7^ here^_is -'knowledge content." So before making further 

analysis, a good tactic.-wotrld be: 

^Identify knowledge courses now in the Navy training curriculum that 
" could be candidates for shipbo.ird training. 

2. Assuming at least equal - and probably superior - training benefits 
from M/F as from paper, determine the cost advantages of shipboard as 
contrasted with shore-based training on knowledge material. 

11;^^ h-^K*^'^^^^ ^''"^^^ analysis could in itself lead to a justification for M/F " 
and shipboard training. Clearly, this would not constitute a full "range of- 
rlt'her than ine;namesl' ^"necessary for reaching a decision to use it 
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EXHIBIT: ETAM EVA'JiAT'nN ^sr viTf,--!^^,V;J n.CMr^ Continued) ... ' • 

If, however, the decision iikiKe-, J.j;n«r:d , as a basis .qv "their decision, some 
range of training in t(V5k-s'sir.s o.^ ipscrfie'J in csuJition to knowledge train- 
ing, then the following £te;«s will nave to i>3 ptrfoni.ei. 

These steps might also be f.er r^^finiu i v tha objectivs: ^ore not merely that of 
accept or reject M/F as a niacium g^, the isa.-t cmount or inforniation that would 
enable such a decision, but tc .wxle a re^ear-.-.h tyoe inciuir> into the possibil- 
ities of M/F for the fullest rcflvic. .:• ,o.3nlia! applicability. Such informa- 
tion would be useful and pe.Sidpr. tsuer.ri^^! a;j f« basH for a system design 
structure that usad M/F or a simi .ar mfr-dlum, ^1st^uct^on■d^puFposes". It 
should be noted that although fzTAifl i';, inlendea to ic Vf.; this kind of purpose — 
as demonstrated on the follcwir,:, pacrH- of asia the performance of this 
purpose will entail more If^bor t' an the „-!ni,,iv.;Ti t^at niiqht be required for pro- 
ducing "accept-reject-stuay fiirt^!-» ■ ,<:rob of ;iecis!cn^r 

The exercise of assessment wi ass'.;-.*;; du" i 0.!.' or loth of she objectives cited 
in the two preceding paragraphs ap;/fcs '.o ths ."iblem. 

(The point should be underlined, r.cwev.-r , ,:'ict -tccme f-m is comprehensive, 
its application is necessarily cxL^nnive. no n:rt"i.?r vih3\ the problem of assess- 
ment may be.) 

Choose Maj or Route;i,_ i;oj^j^<:ri£e-oi'- t "feet (05 .01 ) 

It is expected that part of the jusi if i cation Ur nicrofiche techno- 
logy can be developed f'-i-i d^S;>lw^<^-ii''ift co-.i::- in the printed paper: 
costs of paper, pr1?.t.:.>j, d-hst-ibuiior , .tur-jtje, up.-'ating and obsoi- 
.escenc'3. So om prony ov ItvjiJiry ;;h?,:. i-f instructional 
Vehicles . 

It is also important to dc?tenin ne U.a ra-iga instructional applica- 
bility of the micrcfkhe system, Th<; ob.ie.r. ; vt is to determine the 
teaching./ learning hc-rs of short..-oi--.Ki trrrii.i.-q i-i cl ass rooms across 
^ all applicable cou..'se i.uvceri o i s ;,!^dc cun chs^pi jc.ed by shipboard 

training that can ba achiaved ;-hrc..!c,-, microfiche. The analysis will, 
therefore, -^cten.pt t.o - de.il'ry t-,- .-./fe^^ or classe;- oi knowledge and 
skill that cau ba acquired, in rr in s.:^-:if; specified part, 

through microfiche fjt^tpe T.v spa: i ...etv^yf: -f what instructional 
objectives are Dc-pl :cat5 V. to r,,c:-r, • . - il e-:.p be applied to ex- 
isting courses in ordjr to r.uurci'-/ -..lonciai range of applica- 
bility of M/F on shipLtvru. 

At a later time, when pi .ins hsve D*(.r. :.cmi)le;.ed for computer-assisted 
development, distribuclr-n- «r.'J man«vein3nt of training, and job aid de- 
velopment wi th I-i/F tcciinol oqy, .• liiird r...i> ^.f^ ■ cf as-ieisment^will be 
through the Developi.iant mC Adn.^-i .li^c'-c-' ve Narv.jf.iw-nt of Training 
route in ETAM. The M/F pro ra^n hfx:, uot sl'ive i op;?..i to a point where 
this analysis would now be prc'l cslie-:. It i-. j?xu:ict«d that the 
following cdiegcrjes in tno DiA rout'? ivouTd-ba applicable: ■ 



TAEG REPORT NO. 12-3 



EXHIBIT: ETAM EVALUATION OF MICROIMAGE TECHNOLOGY (Continued)... 



• Developing training requirements f ran skill requirements 
descriptions. 

• Selection or devising of training modules and modularities. 

• Designing the content of instruction. 

Although microfiche as an instructional vehicle does not have proper- 
ties directly applicable to these D&A, issues, a semi -computerized 
subsystem which produced and selected microimages can be devised 
that would enable this medium to be more attractive than Daoer equiv- 
alents. K K H 

COninent 

The present assessment will extend to the use of M/F as a substitute 
for paper job aids such as technical manuals on shipboard. The assess 
ment will assume operational feasibility for the substitution of M/F 
displays for paper, supplemented with printer-copiers of M/F images 
where paper copies are essential at a given time. The operational 
justification for this assumption will not enter into the present 
assessment, which will, therefore, be limited to the economic oppor- 
tunity in substituting M/F for paper. 

-Vehicle Type That Fits the Innovation (05.03) 

The microficjie and its viewer are applicable to the full-range of 
static graphics used in instruction. It can, thus, display any con- 
tent shown on paper, foil slides, or static images on a display 
tube. Fiche images can be used to project enlarged images for group 
discussion or classroom presentation, thus displacing slides and 
foils. 

Npte that an economic assessment could be made at this point: 

a. Determine present costs of all shipboard training, demon- 
stration and briefing materials contained on paper, slides, 
charts, etc. 

b. Determine costs for microfiche equivalents. Assume a rea- 
sonable requirement of paper duplication -- say 10% of the 
total. In both cases, costs should include generating and 
editing the content, updating it, distributing it, storing 
it, and accessing it selectively. 

c. Compare the two sets of costs. A preliminary justification 
for M/F may appear and be more than marginal. In any event, 
a cost ratio between M/F and other media with equivalent 
function will be generated. 
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Type of Training Objective (05.04) 

In general terms , microfiche can be used for the learning of: 

a. Reference knowledge, 

b. Knowledge that is task-specific and enabling to the 
extent that the knowledge can be acquired, retained, 'Sind 
applied as a verbalized, cognitive content, 

c. \^Some task-skill fonnats at preliminary stages of compe- 

tence, 

d. Some skills that are primarily symbolic transactional, 
such as filling out a requisition or a duty roster. 
These may be treated?i.as "task formats", however. 

The aforegoing identifications mean that specifics will be sougHt 'in * 
the assessment routes leading to Reference Knowledge, enabling Knowl- 
edge, and Task Fonnats, 

Select Vehicle Properties (05,05) 

At this point we become concerned with displaceable costs in the pre- 
sent inventory and usage of M/F alternatives in current training 
courses. 

Since the "Instructional Functions" in 05,05 are roughly equivalent 
between *the M/F technology and the alternatives it is displacing 
(paper, slides, etc.), these functions are not pertinent to this 
assessment and can be disregarded. 

The "Utilization Functions" that do apply for comparison purposes 
consist of: 

Prograrrmability of instructional content. 
Portability and storage. 
Multiple usage. 
Maintenance. 

Confining instructional capabilities distributed^ among several 
n^di a/devices. 

Modularization of trainingp content for more general appltC:ability 

Cost advantages and liabilities may ar1.se in each of these categories 
of c6st assessment. It should be recognized that raw dollar estimates 
of saving may present an incomplete picture of value. For example, 
it is common practice to assess the cost of building space, on a per 
square foot basis. But on shipboard, construction costs per square 
foot of floor space v/jU^be an incomplete evaluation of savings. The 
draniatic e)jample is cubic foot space in a submarine. The assessor 
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may, therefore, want to get the help of appropriate operations people 
to develop a "functional multiplier" for the unit of objective cost. 
Thus, on a given type of ship, the construction cost per square foot 
may have a multiplying factor of 3 for space saving. But because of 
considerable idle time of shipboard electronics maintenance men, the 
normal "maintenance cost" for simple instructional devices might well 
be .5. Obviously, when a multiplier of this kind is used, both the 
multiplier (and its rationale) and the cost factor as Objectively 
determined should be presented to. the decision maker. 

The portability of a compact, lightweight M/F. viewer, plus the instruc- 
tional and reference context in a few square inches of M/F cards, can 
be assessed in two ways. One is in the readiness with which a rela- 
tively small stock of viewers can be borrowed, used, and returned by 
a number of trainees. There is, thus, high opportunity for much usage 
per instructional entity or unit. The second^point of view is that 
the combination of compactness and low cost makes it possible for all 
learners to have a copy of the device and instructional materials in 
their private living quarters where it is continuously accessible to 
the spare moments of the learner, as well as for more extended periods 
of study time., 

(Incidentally, books may be printed in M/F, and these may include 
reading for entertainment as well as study materials. This may add 
to M/f justification.) 

Specifications for Searching the I nstructio nal Vehicle Inventory (05.06) 

In this example, the search s,pec1fi cation is simple: it consists of 
"static graphics" as defined in the ETAM context. 

In this case, however, it is simple to become more specific in identi- ' 
fyirfg the names of the relevant media. These are: 

All printed materials used for instructional purposes except 
student evaluation tests. 

All slides , foils , e.nd other-' single image transpardencies for 
projection puV'poses. 

All charts, diagrams, tables, pictures, ^ ^ ' 

Manuals of operating instructions if they are used tor teaching 
purposes. If these manuals will be used on the job in M/F form, 
there is a clear cost advantage for using them in M/F training. 
If this is not the case, a separate assessment should be made 
that compares a redundant M/F version for training with paper 
manuals used both for training and on the job. It should be 
recognized that the shift fran M/F to paper will call for some 
adjustment by the incumbent, but this may be transitory, espe- 
cially if there is a common indexing structure to the two versions. 
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Identification of Relevant Inventory of Training "Vehicles" (05.07)/ 

The listing of types of static image representations culled out of 
the total inventory list will identify the contexts of application 
and use. Of the total list, a number of items may be judged inappli- 
cable for M/F treatment. For example, a class schedule which directs 
where an'd when .a student will take what classes should obviously be 
on paper that the student can carry with him. Some, condensed "pocket 
guides" of definitions , general structures' of relationships (such as 
the schematic of a procedure or a device) should be on paper carried 
by the student for rapid reference either at odd moments 'or in con- 
nection with study. 

After making a number of detai led examinations of items p the list, 
the assessor may decide, on the basis of the sample he has examined, 
that a reasonable est-^mate^is that 955^ of all the applicable types 
of a given medium (say, paper print) is subject to M/F application 
and apply this formula to the remainder of isB.e relevant "^i tems . The 
potential error in the final assessment contributed by such estimates 
will probably fall into the cracks of estimating unreliability. 

Identify Benefit Pattern for Instructional Vehicles (05.08) 

Separate comparison breakdowns of unit costs would be prepared for 
instructional materials on paper p on. slides, foils, charts, and for 
M/F for similar m'aterials. . 

Cost data would be obtained from -samples;' typical of each- medium 
according to the following: 

Preparation of the image content to a process-ready state. 

Image j^eproductipn , singly and in groups representative of a 
collected unit such as ci booklet, or training element. 

Cost of raw materials; spoilage and wastage factors,/ 
Distribution costs. • ^ 

* 

Update (with typical examples) costs to process, distribute and 
enter jnto user image files. ' , ' 

Inventory maintenance costs: storage, handling, obsolescence. 

Medium holding^and handlingmaterials: book binding, slide 
mounts and holders^ etc. 

Associated display device costs per unit of use; e.g., M/F 
viewer or slide projector. 
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To these coVt comparisons will be added benefits data frm Task Train^ 
ing analysis. 

Identification of the Learning Objective (05. 10)^ 

Those listed in the Task 1 Description of the Microfiche Technology 
will be sustained and rei^eated here. In genei^al > they are: 

Reference knowledge. 

Enabling knowledge. ' ^ " ■ 

Some task formats to limited stages of learning. ^ 
Some transactional procedures to limited stages of .learning> 

Further specifications of objectives will be made in the respective 
,topics cited ^bove in later sections of this analysis. 

Preliminary Evaluation of I mpor tance of Benefits (05-11)^ 

Further examination of the potential benefits of the technology as 
applied to shipboard'training has sustained the evaluations of im- 
portance made earlier. . ' . .. 

1. The training objectives, if feasible in^ thts/environment , ' • 
are at least of moderate imporjtance and cou^d become, if 
sufficiently widespread, of very. subst^nti al importance . 

2. The magnitude of the benefit, if microfiche became a key 
factor in<the practicality of widespread shipboard training 
in job knowledges and formats^ could" assume vef\y substantial 
importance to. the Navy. This is on the assumption that some 
X percentage (to be estimated) of all shore--based training 
could be shifted to .shipboard' training. ' " 

Application of Microfiche to Skill or Knowled ge Trai^ning^ (05.12) 

Assessment will^be made both in the knowledge branch and the skills 
branch of the 'analytic procedure- 

Task Structure 'Elements in Skills Releva n t to M/P Instruction (05.13) 

Thfe Si^suitip.tion is that the M/F device or viewer is advanced manually 
frame by fram'fe. Advance may be r^J atively random with respect to 
frames on a cjiven card. The content of the frames ^on a card may 
be designed sp that the center row acj^^^s-s the card contains the* primary 
level of instruction sequence. If th?j ^student requires more procedural 
information at a given frame in the primary row, he moves to the frame 
a row above or below the primary frame. Each row contains \a standard 
category of information to supplement .the pnimary row. .A g\iven row 
may contain workbook information, or l;est .questions for the primary 
frame in its co.lumn. Depending on his' needs, the student can, obtain' > 
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supplementarinstruction at any point, and test himself. (These pro- 
cedures have been described in an R&D report, and demonstrated to be 
effective.) 

The question in this topic is what skills or components of sl^ills can 
be learned merely from static displays, without timing constraints, 
and where the student expresses his response output symbolically and 
must himself determine, from displayed information, whether his re- 
sponse was adequate or inadequate. ^ (Separate analysis will be given 
•to knowledge and task format objectives.) 

Goal-Image . These can. be learned to the extent that the goal-image 
can be displayed realistically in the form of text, diagram, .or pic- 
ture. The image represents /the appearance of a state of affairs when 
the task is properly performed as contras'ted with outcomes when the 
task is improperly performec^. The student learns to make the discri- 
minations. In the process of M'nstructibns he may be given analytic 
guidance for making these judgments, but ultimately he may have to 
recognize all-at-once whether the picture is correct or incorrect, 
and specify what should have been done, or what should now be done 
to correct the goal-pi cture . 

Some examples of goal-image's learnable by M/F are: 

■ ^ . • ... 

Correctly filled-in questionnaires of a g.iven kind, or other 
/transactional document|s; dietary menus. 

Sighting pictures. 1 

Wave shapes shown on test instruments^. 

i 

Diagrammatic and verbajl descriptions of battle plans.'. 

Except for the first item, hll of these are examples of lear'ning that 
call for Identifications and, i,n some cases. Interpretations. The 
first example in the list, Although legitimate^ to Goal-Image, may be 
equally legitimate as an instructional factor in "decision-making" and 

fact that expjnples for Goal-I,ma'ge teachabl 
her more inclusive categories of learning 
abandon this as a search descriptor, and 



"constructing" formats. Th 
by M/F can ^dissipate into o 
operations suggests that we 



Scan-Detect . Orjjy the scanning-detecting in static images will be 
applicable anjl^his element should, therefore, probably be included 
as members of a set extending to Identify. The types of task content 
for this category applicabl^ to M/F clearly are maps;^ charts, blue- 
prints, semi-symbolic representations such as radar 'di splays , and 
materials used in photb-reccjnnaissance interpretation. Knowing the 
applicability^ of th^ese examples of^content should help the investi- 
gator judge whether to accept or Reject the hits made by the Scan- 
Detect-Jdentify search on job tasks or on training course d.escriptions. 
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The large' number of training and test exarripl*es that can be put into 
small/ physical space on M/F for individual use make this an excell- 
ent medium. This value is augmented by the student's ready access 
to Supplementary guidance for each problem presented. Useful skill 
leyels in se^in-detect-identify are acquired only when. there is the 
speed and reliability of performance that can be attained only by 
ipny hundreds of practice examples. This is no|t a trivial training 
/function where it is applicable. f 

Identify .' Part of this task function's applica})ility has been 
treated in the Scan-Detect topic. What is addiiionally applicable to 
the learning of task information consis^ts of: 

Nomenclature: narijing and identifying work objects, processes 
. and characteristi cis j specifying or -jrecognizing their location 
and physical relatjionships to other work objects. 

The learning of task nomenclature is often not a trjvial operation aind 
can be a useful substrate to learning the procedures that apply to the 
work objects denoted by the nomenclat^ife. M/F and a manually operated 
device may not be ideal for learning this kind of content, but it can 
serve ^adequately , especially if supported with effective student test- 
ing operations. 

Interpret . Unless the task is time-driven, the capaci ty f or M/F to 
deliver hierarchically organized and accessible infonriation should 
make it useful for training up to at least intermediate competence 
in the 'interpreting" demands of many tasks. A situation may be 
displayed, and the student is required to interp^x^t its significance 
eitffer as to the circumstances that caused the pattern of conditions 
to appear, or in the judgmental aspects of selecting a usefu-> task 
response. It is perhaps more useful for the student to be exposed to 
a largi? number and wide variety pf situations requiring interpreta- 
tions, than to engage him in profound analysis of a limited number, 
of examples. A substantial aspect of his training and performance 
is in his ability to hold in mind the context of infonnation upon 
which the interpretation is based. 

Examples that would apply to M/F instruction are: 

A pattern of symptoms in a device, where its interpretation could 
result in determining (a) the cause of the symptoms and/or (b) 
what capabilities the device still retained for operational per- 
formance. / 

A pattern of enemy actions is described from which the enemy's 
intent and objective is to be inferred. ' » 

A pattern of behavior data from an i.ncumbent from which his 
capabilities and motivations are to be inf erred » 
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A pattern of signals is apprehended through a noisy channel 
from which the entire message is to be reconstructed. 

A pattern of symbols is displayed on a map, or other type of 
diagram is presented from which the actual physical charac- 
teristics of the objects symbolized are to be interpreted so 
that they would be recognizable if present. 

The foregoing would be guides for identifying training objectives 
and their contexts that might be identified under "Interpretation" 
but require additional judgmental criteria for selection as candi- 
dates for training with M/F, at least to intermediate stages of 
competei ^e in the task element. 

Interpret s Where the meaning or significance of a pattern of cues 
must be translated into an inference about a probable state of affairs, 
cind there is little interaction allowed in the task itself for speci- 
fic inquiry for further data, M/F can provide useful instruction. But 
in the contexts of relevance to M/F capabilities, the content of such 
instruction can more usefully be characterized as "task-enabling 
knowledge." 

What will be excl uded from consideration for M/F training will be 
such meanings of in.terpret as: Interpret from one language to 
another, and Morse code interpreting.. Although it is possible to use 
a static medium for learning many aspects of a language translating 
skill, and perhaps^ especially if it is coupled to-an audio input 
function to the student, the M/F medium is apt to be highly Ineffi- 
- cient, (If there. are empirical demonstrations to the contrary, the 
data should have primacy in guidance.) 

The interpreting function draws upon task-enabling knowledge, so we 
can expect overlap of instructional content that is drawn from a 
task requirement using a term like "interpret" or deduce or figure 
out or similar expression with content that is drawn from descrip- 
tions of job and task knowledge, especially where the criterion of 
"enabling" has been used as a filter of content. 

Note : The brevity of treatment of a topic, such as the 
present one^ should not be misinterpreted to mean that 
little or unimportant ins.tructional material will be 
derived from it. In this case, it is the contrary. In 
Q fair number of tasks there can be substantial instruc- 
tional content derived that is applicable to "interpret." 

Procedures Performance . Microfiche technology with the simple 
viewing mecTiamsm postulated for this assessment can train in any 
procedures which have been amenable t\ conventional Programmed 
Instruction. ThuSp any PI on a paper niedium could be readily trans- 
lated into the M/F medium with somewhat greater flexibility in sequen- 
cing and material than is practical (in terms of the student) with 
paper. 
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Note that "formats" for procedures will be treated in a later section. 
In this section, the concern is about instruction in skill learning. 

M/F lends itself to instruction in procedures that are restricted to 
symbolic materials and symbolic operations. It would be somewhat 
less efficient than, say, a computerized capability of answer analysis 
and branching to specifically suitable remediation. (One can be 
overly preoccupied with this kind of efficiency and neglect comparing 
the real benefits with the very real costsi) Examples of symbolic 
materials and operations are: 

Learning and applying- calculational rules. 

Navigational calculations with various kinds of input data 
as given. 

Performing statistical analyses and interpretations. 

Filing procedures; setting up files. 

Most clerical transactional procedures (see the analytic 
definition of "transaction" in the Appendix A section 
called Task Formats as ct guide for applicable task 
s tructures here.) 

Decide . At least preliminary training (at the decision format level) 
can be given in classes of decision that are wel 1 -structured (all 
variables relevant to the decision class readily prepresented) , 
where the apprehending of the problem information is not dependent* 
on absolute time values, and where the d-ecision itself is not time 
dependent. Furthermore, because of the limitations of the medium 
and device, the problem information must be presented in a form 
where the variables have already been symbolized (the learner does 
not have to identify the variable and the value in a complex 
perceptual environment) and usually the learner has only to select 
from a restricted set of response alternatives. 

Static display media, while able to give limited practice 'to the 
judgmental aspects of linking a problem situation to a decision 
response, may give extensive "task-reference knowledge" by giving the 
learner the impTicatioos of selecting various choices in a given kind 
of situation. Acquiring information that enable^; judgment in 
decision making is often difficult in real life because the incumbent 
frequently does not see the outcome of his choice, or if he does, so 
much time separates his making the decision from learning the outcome, 
that learning is ineffi c^ier'c and unreliable. Thus, although the 
learner may, with this ^nedlum, acquire only limited levels of skill 
in actual decision making, he can acquire verDal and conceptual 
information that will assist in that skill. In summary of this 
point, he can be given practice enabling him to anticipate outcome 
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probabilities from selecting a given choice, as well as in the 
learning of the variety of options he may have available from which 
to choose. These might be called "task-enabling knowledges" for 
decision making. 

Some decision-making task formats may be learned with the M/F meduim 
and the simple display device. 

A qualification should be made here. The development of instruction- 
al content that will generate useful practice and learning' even in 
the preliminary aspects of decision making (coping with uncertainty 
in problem situations varying in their degree of explicit structure), 
requires a high level of teaching art and insight into human decision 
processes. This level of talent is likely to be rare. A practical 
decision in assessment might well be to forego "decision making" as 
a task element for which M/F and the simple display device is a 
fruitful application. 

Construct . The same arguments developed for Decide apply here 
except that the limitations of the medium and the dependence on the 
art of thejnstructional content designer is even greater. 
"Construct" is not a promising candidate except for perhaps very 
rare tasks. The learning of task-reference knowledge to support 
skills in a given kind of conslructing can probably l}e done with 
M/F, although not with ideal efficiency. 

Microfiche and simple viavers hold little or no promise for instruc- 
tion in the following tas^ elements: Tracking, Motor Performance. 
Interpersonal Interaction. Recall of Task Cycle Infonnation , or 
Recall of Enabling Infonnation. The last two task elements are 
processes subsumed under other rubrics: M/F does not offer as a 
medium any special properties for either typo of recall that are 
not shared or utilized by competing rnedia. 

In sunmary, we have the following descriptors. that, with some 
qualifications in all cases, can be used to search the data base 
description of cour?es that include skill (or "practical factors") 
training. - ^ r / 

Goal-Image: heavily restricted to tasks dealing with 
vertal fomiats and content, soma perceptual skills in 
differentiation, and diagrairenat^c representations of 
task content. 

Scan-Detect-Identify: cognitive content in maps, charts, 
blueprints, semi-symbolic representations such as radar 
displays, and photo-reconnaissance. 

Identify: nomenclature, identifications and locations of 
work objects. (This item will be repeated in "Enabling 
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■"Knowledge".) Also tne identificatTon of patterns of 
elemeiits that in combination become an identifiable 
entity. 

Interpret: enabling Knowledges that enable the operator 
to determine the meanings implied by patterns of task 
cues. (Note a distinction between identifying a pattern 
and interpreting what the pattern means). 

Procedure Performance: in skill learning, restricted to 
early and preliminary levels of symbolic tasks and materials: 
calculations, navigational calculations and logic; clerical 
transactions as examples. 

Decide: at least preliminary skill training beyond the 
level of decision formats, but in tasks where. the actual 
problem information is presented symbolically as cogni- 
tive content and not time-dependent such, as the decision 
to abort a landing on a carrier, or where the context of' 
problem and resource is so complex that it cannot usefully 
be presented in static displays. Best bets are clerical ' 
transactions ^which demand "judgment" in the face of some" 
Indeterminacy and uncertainty. 

Construct: only task-enabling formats for constructing 
a class of entity. 

r 

No additional task elements seem appropriate to the M/F medium 
except as they have been already treated in other task elements' 
or in the enabling knowledge. 

Identify Relevant Jobs and Tasks (05.14) 

In the present assessment,' this step will be skipped. The reason is 
that the major targets for M/F will be "knowledge" courses, and the 
"knowledge" component in practical skills courses. 

If, however, a job-task data base were to be learned, the task 
element names selected in 05.13 would be applied with the following 
restrictive specification: 

Equipment and objects used in che task : "symbolic". 

Reference, information used in performing the task: "symbolic; 
AND/OR static perceptuaTT^" 

Relevant Stage of Learn inci (05.15) 

Different classes of learning objectives applicable for microfiche as 
a medium and the simple viewer ~~ can be learned to somewhat 
different stages from the point of view of their contribution to task 
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performance level. Refer to the objectives selected in Subtask 05.10. 

Reference Knowledge : although these can be learned to reliable 
levels as knowledge, practice in the context of the task itse^lf is 
required for their effective integration into performance. Thus, 
with regard to a task performance criterion, reference knowledge will . 
be learned " to the verbal leveV . 

Enabling Knowledge : the same applies here as for Reference Knowledge. 
Stage of learning will be to the verbal level , except in the case of 
Tr^ansactional Procedures, treated below. ^ 

Task Formats : with respect to symbolic or perceptual tasks, the M/F 
training could take the student to " Performing Task Components with 
Guidance ." 

Transactional Procedures : the qualifications made in 05,13 should be 
recalled here. The tasks that fit the qual if ications can probably be 
Ifiarned to the level of " Doing the Entire Tasfs/Job Procedurally : 
Barely Aebeptable MasteryT ^ The student's reaching thlT level wi 1 1 
depend, among other Things, on the degree , to .which the practice con- 
tent has adequately sampled the range of problem situations in the 
real job environment. 

ATI Procedural Operations : although the initial objectives' for M/F 
usage did not identify refresher training , M/F could be given on 
jobs currently being held by the incu"mbent-student and especially 
over knowledges and procedures that may be rarely used. Refresher 
training may also be given to students who have completed courses of 
training but who have not been assigned to jobs in which they can 
exercise that training. The present assessment wiOl not include 
refresher training in the analysis because its inclusion could cloud 
the main issue of M/F as primary training. But its shipboard use for 
refresher training could be a strategy for widening its usage to 
primary training, and thus gain acceptance progressively. 

Refresher training would be applicable to al 1-. incumbents in their 
present job positions. It would also apply to all incumberlts who 
had training in duties into which they had not yet been assigned. 

Identify Relevant Course s (05.15) 

Som labor saving strategies may be used in determining courses and 
course content applicable to M/F instruction. , Some of the following 
repeats advice offered in the introduction to Task 5 

1. Courses that are primarily knowledge and theory background 
. for practical skills training. Except for testing and 
student evaluation, virtually all instructional functions 
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provided by lecture and paper can be taken over by M/F. 
Whether the motivational functions of the classroom 
environment (such as they are) can be replaced in the 
self- teaching environment would require empirical Inquiry. 
The equivalent in classroom course hours (which normally 
does not include self-study hours) in M/F study hours 
might need to be determined empirically in any benefits 
evaluation, although student time of shipboard given to 
study may have a different cost and value than student 
time In shore-based training. 

2. Skills courses potentially applicable to shipboard training 

Training Units Within Courses (05 J 7) 

Within skill courses, the following training units or objectives will 
be applicable to M/F instruction. 

1. Any type of knowledge as identified in Subtask 05.19. 

2. Procedural and transaction- formats as defined in Subtask 
05.23. 

3. Nomenclature and identifications specific to task and 
environment. 

4. Task structure elements as summarized at the end of the 
writeup for Subtask 05.13 in this assessment report. 

Procedural Comment. A short-cut for making estimations can be based 
on sampling procedure. Subdivide all skills courses considered eli- 
gible for shipboard training into groups that deal with (a) opera- 
tional skills wi4:h equipment; (b) maintenance jobs; (c) service and 
miscellaneous jobs, Including clerical; and (d) Command jobs. Within 
each group, choose at random half a dozen course descriptions at-^^the 
most detailed level available. Make an analysis, objective by 
objective, of each item of training in the course description that 
fits the applicability to M/F according to the search criteria estab- 
lished In Task 5. Make an estimate of the ratio of the training 
hours of M/F material to the total course hours. Thus for one 
course 1n a group, the M/F applicable content may be 25% 'of the total 
hours; for a second course it may be 22%; for a third 28%. If there 
Is relatively 1 ittle practical variation among the samples of these 
ratios for the group of course, choose the representative value among 
the sample ratios and extrapolate to the ratio value for the entire 
set of courses in that type of course. If the sample values fall 
within a 10% range for 90^- of the* sample inemb^:rss it would seem safe 
to choose a mean or median of the sample values as representative of 
that class of courses. 
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In some job and training specifications ^ the Navy has already made 
a division between knowledge content and practical skills content. 
Thus, in the Personnel Qualification Standards, the 100\series and 
the 200 series are directly applicable to knowledge trciinina-, hence 
to M/F. • , C ' . 

In the format used for the Navy's "Occupational Standard^^orkshop 
Worksheets," the knowledge factors are separated from the Practical 
Factors (skills). 

Where job specifications of these kinds are reflected in the specifi- 
cations of training objectives, the identification of M/F appHcab1«^ 
training is tedious rather than complYcated. 

As a course applicable both to shipboard training and to the M/F 
medium is identified, the projected numbers of students for that 
course is noted. These numbers will be used to put scale into the 
magnitude of the potential M/F application. 

Assessing the Benefit Pattern (05.18) , 

The two innovations that are jointly to be assessed consist of the 
substitution of M/F medium, plus simple, viewer , for paper in 
instructional materials --^w1th the unassessed by-product in M/F 
substitution for many Navy technical manuals now on paper — and *:he 
substitution of shipboard training for a large proportion of shore- 
based training. In both of these factors, the major benefit to be 
expected is a savings in costs- In part at least, the ability to 
reduce the bulk of training material from that of paper to- that of 
M/F, makes it practical as a "necessary" condition for shipboard 
training. The range of training (in terms of courses and jobs 
affected) and the Importance of what can be learned with it makes 
shipboard training feasible . 

But shipboard training will be shown to be practical (assuming that 
conflict with operational exigencies has been adequately managed) if 
cost factors for shipboard training compare favorably with costs for 
shore-based training. Making this comparison requires invoking the 
cost model for shore training and an equivalent cost model applicable 
to shipboard training. 

A factor not previously examined about shipboard training is that of 
supplemental devices, including samples of operational equipment, for 
training jn actual skills (the practical factors) for which knowledge 
and format learning is merely a precursor. Either such skills- 
training equipment is on the ship, or the student must acGuire the 
practical skills in shore-based schools, ov the "skills" factor will 
be satisfied (for advancement purposes) by verbal learning, including 
descriptions of procedures. 
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The most realistic assumption is that only those skills courses will 
be taught on a ship for which there, are copies of operational equip- 
ment on which students can learn skilled performance. ,. But this 
assumption In practice has costs, possibly including operational 
costs and risks, which complicate the shipboard training picture, 
beginning with acceptability to the commar.der of the ship. 

Let us assume that cost comparisons have been performed and qualified 
by judgments of probability of achieving the cost-benefits of M/F. 
Let's now make a pass at the questions in 05.18. 

1. Reduction in training tinie . None expected. If M/F 
increases training time as compfirGd *wi th shore-based 
training, the costs are not relevant to any budget. 

2. Reductions In attri tion rate/- There are somiswhat different 
factors that, a prion , seem' to v/ork for and against • 

. attrition rates in shore-bcisea conventional training 
and shipboard training. Shipboard, the trainee works .at 
his own rate, and there is little or no penalty if he 
takes twice as many study hours to complete an objective 
to criterion mastery as some other student. On the other 
hand, he may have less direct association with instructors 
on shipboard who. could help him past a learning obstacle, 
or to motivate him individually to keep on trying. All 
together, one set of factors seems to cancel ou.t 'the other. 

3. Reductions in stude nt aptitude. Because there is little 
expense (as compared" wi'tFTsTiore-based training) in - 
committing a student to shipboard training, it is feasible 

•■ ^ to lower the aptitude levels for shipboord training and 
./ take a greater chance on the student':; success in com- 
pleting the course on the assuniptionr- that (a) aptitude 
tests have substantial standard errors, of prediction and 
(b) the student can take more time to complete the course 
requirements. This po>icy ignore^ ^f.ne possibility that' 
there is a relationship between ^^n 'aptitude score and the 
L|lt1mate level of the incumbent's skill no matter how much 
■•training he was given. Any t^stlniate that student aptitude 
could,, in generals be scitiewnnt reduced because of the M/F 
, shipboard capability, would have to be close to zero bene- 
fit. 

There is this proviso^ ho^A'ever. It has been said that in 
many cases the difficulty of the cognitive (knowledge) 
materials in training such as^ for example, "Theory of 
* Operation," 1s one inajor oasis for many student's attrition. 
If success in sucn knov/ledges has a low correlation with 
eventual success o». the job, then any training procedures 
which, help ti\a student get past his fomial knowledge 
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requirements will increase his opportunities for getting 
into practical training srid, thus, to the job. If this 
argument is justified, then the flexibility offered by 
individualized paths through M/F 'would lower the aptitude 
rsqui reman t, perhaps by substantial amounts. 



4. Supplemental benefits. Shipboard training may have highly 
useful side-benefits to the shlo as well as to the man. 
There is at least indirect psyche! ogical justification for 
the belief that "learning activities" and "job activities" 
should not be separate blocks of activity, but that the 
learning process and the motivation to learn should be 
continuous in a person's lifetime. The intellectual 
struggle to learn something new during prescribed hours of 
the day m®y well reflect in greater alertness and inquiry 
in the hours spent on the job, even though the content of 
what is being learned differs widely from the content of 
the job. Opinions of this kind remain to be empirically 
validated across wider populations than merely the pro- 
fessionals where, in industry, this assumption is being . 
applied. People take courses as a part of their workday 
and presumably some of the stimulation rubs off into their 
work. 

Like many really important values and "benefits", this 
stimulus to learn" is not susceptible to quantitative 
assessment. In degree, il wauld be applicable to all job 
:incumbsnts who were also students, and apply to all jobs 
and tasks. It would have ac least a t.ioderate importance to 
the quality of perfonriance of the NavylfnssTon , the morale 
of the crew, and presumably of the reenlistment rate of 
desirable personnel . 

This factor should probably be stated as a qualitative plus 
to the decision maker rather than in a quantitative format. 

Type^ of knowledge (05.19, 05.20 and 05^.21) 

Microfiche as a medium and the simple viewer as the device, supple- 
mented by periodic performance testing with other means, can be 
applicable to the learning of any type of knowledge for which the 

ihltTLi\ Ir^^'} l"" ^^t?- *yP^' °f knowledge learning, 

^nrJh!^b?fi ftr'^uP^PfJ^^f arrangements that can be more efficient , 
But the skill with which the instructional content is designe d, dis- 
piayed, and sequenced may be- more important than any combination of 
other factors for efficiency in lear.nng/and effectiveness in reten- 
tion §nd application.,. . 

All classes of Reference Knowledge (00.20) and Enabling Knowledge 

[.A JL^''^ appl cable. .The more significant classes of knowledge 
under these rubrics are: 

o J 
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System p.urposes . 

A. 

CoBtexts of operation. 

^ * Theory of operation. 

Enabling facts and prescriptive rules. 

Concepts aad principles. , 0 

Nomenclature, identification and locations bf work objects. 

Procedure descriptions. 

Typ6s Of Task Formats (05.23) ^ 

The following types of format are susceptible to learning from the M/F 
medium and the simple viewer. This combina^tion is somewhat more flexi- 
ble for branching purposes than is paper, but far less so than a com- 
puter-driven display terminal. The branching requirement is essential 
for enabling the student to try out various alternatives, and to deter- 
mine the* implications of Vcirious alternatives, as well as to present . 
to him contingencies that may be associated with a ''standard" situation 

Procedural ^ Formats : the general sequence of actions whereby 
something gets done. ' 

Tran-saction Formats : preskill and skill learning in performing 
more or less routinized, well -structured transactions (often of 
a clerical type) that involve filling out of forms, or trans- 
lating information from input forms into 'output forms with some 
processing judgment. 

i>r Decision . Fonnats : learning the variables of problem and re- 

source to keep in mind and optimize in choice-making activities. 
Extent to which the student's management of "branching" opera- 
tions in M/F enables skill development in use of the format on 
task samples will probably depend on the specifics of the deci- ^ 
sion-making task and information content. Decisions of the 
clerical ty[ie associated with transf?ction formats, identified 
above J can almost certainly be done with M/F because the task 
usually presents limited alternatives. 

Conclusion to the Assessment Exercise 

In many ways, the problem of M/F and shipboard training is an example that is 
not representative of the primary objective of ETAM which is rather consistently 
'^j^ointed towards practical skill training. But perhaps this is why it may be a ' * 
useful test of the procedure. . • 
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?nn?J hP^lJf '"^''^^'^ ^^P"""' ""^^^^y assessment outcomes which' 

0? ™ J S o'!/ very few pages - but the lines of thinking, with the format 
of ETAM .as a guide to the conclusions made. It should also be noted that [hp 

^ask f J?;?/^'?'^^!^ ^^"^^^'^ °f ^^'"^^t "° empirica data Jiven i" the 
Is a ledlum Tleast a? lnl'''''^^'''r f '^^^^ that U coi d erve 

wSaJe^e^J^ltJL'L^ug^^t^Ll llafntrwSt^S^^p^S^: static presentation medi^a) fo. 

One Intent for ETAM was that assessors would be led t6 think*-constructivplv 
? tn' ^rnnr?''' might -otherwise be ignored. E^ertl'oug 'Jh' ItaS-^^ ' 
ofTmJ2? «S were created over a number of -months and with the help 

sfqnIflSnJ nroSr^ r 1'^'^°'%'^ ^^^^^ the present ?omat wa sa^ 

wi?r:i^i?fosr^sh^s?e%^^?s^^^^serJ^t]^h'°^^' '''' °^ ----- 

An attempt has, also been'made to show examples of by-passlnq procedural c-tpnc: " 

P oced'^s'empSa^iL' t^li:^ J:^''''''' 4' ^y-P-sin^" A?th^^Tth'e\lJM'^ 
oblectlves iTrllll ,f ^/J^essors make analytic short-cuts when their 

th s procedural nexib??^ ""''^ ^''''^''^ con^uni eating 

sive than ?rrinm-rfH 1 Ju^^^''- ^'^^"^'^ P^'nciple. be more exten-' 

s s itM no^?yhi! ? "'""^ the decision which is the objective of that analy- 
thrdedslon's°hoi?i'be°' ^^^^^^ ^^^^^^^ t° ^"^'Veal what the nature of' 

etas 

Another major value in this forfn of assessment is that, by being structured 

hpJtl^'/f ■'"'^u' ^"^ ^^^'^^■"S' be reviewed b; other wh^marbe 
better informed or who have additional insights. These may contribute to 
effective modifications leading to better decisions about acceptance reiec 
tions, or "more studv." Rut nprh;,nc 4. ..u =r , ^^'-T^J^^ 



t1ons. or ...o^e study... But plnhaprjjst ^r^^ponan^r^h^^adSe^^^San'^^r 
cs may lead to the superior use of an "innovation." or better 
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tions,. and insights may 

safeguards against its misuse. Thusrtte-ass;ss„B,,r'pr;cesriis"elf ^ake^'con 

abl^fn ;h»*H-"'%'"''!?'°**' ^''""t^"" snd training under ond tlSns"a»o? 
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SECTION ,VI . •■ ' " . ■ . .. 
X ETAM VALIDITY AND UTILITY 

The validation of a procedure can be split>.into several paiffs^ One is 
'the extent of flexibility and adaptability of the .procedure to\ variety 
of conditions and objectives. A second is whether the procedures can be 
followed, and with what difficulties, uncertainties and pitfcillsS The ■ 
third aspect of Validation is the evaluation of how good the results of 
following the procedures can be. . ' 

FLEXIBILITY ^ . 

I-t is unnecessary to crank an assessment through the entire ETAM procedures 
in order to present data to the decision process. The* description of the 
innovation, as structured in Task 1, may be' passed directly to Tastes s3 and 
4 where they-^can be formatted for presentation to the ciecision maken. Wher- 
ever -the pattern of the benefits projected from a proposed innovation suf- 
ficiently outweighs the costs and liabilities, the combined patterns may by- 
pass the remainder of the data collecting and analysis steps, and be pre- 
sented to the decision maker. - • - 

Task 5 assists the assessor to probe Into the ful^lest^potential range of 
effect of the innovation in terms of extent and boundary limits. But if 
at any stage of inquiry, a suffi cie^nt range of effect has been determin'ed 
to justify a benef i ts-to-cost 'decision , further analysis can be halted and 
the ca'sQi can be presented to the decision maker. If gross-, overall estimates 
of a benefit are justified, (say for all of a* given type of course) it. is 
unnecessary to undertake analytic work for each individual counse and sum 
the totals. If ,^ however, a decision is made to accept the innovation and 
at the same time' specify a prograin for implementing, the decision in prac- 
tice, "the full-'^cale analysis in Task 5 should be made for that purpose. 

Task 3 in ETAM poses a number of key questions to be addressed to the pro- 
posed innovation. Even one crucial , negative answer can justify terminating, 
further assessment because the innovation has failed to meet a necessary 
condition to acceptan-ce: ^so why continue further? 

The assessor is invited in Task 8 to^ anticipate the kind of data presenta- 
tion and analysis that the decision makers will want. Some will prefer 
emphasis on the finanpial analysis developed -in Task 7, others will con- 
sider the output of the benefit-cost analysis in Task 6 as the more impor- 
•tant input to the decision process. Furthermore > the decision maker may 
prefer to ignore the outputs of the analyti cmodels and prefer to base his 
decision on the picture of the* raw benefits, costs,- liabilities and proba- 
bility data developed in Tasks 1, 3 and 5; as well as on several alterna- 
tives which may have been developed in" Task 2. 

This flexibility in ETAM procedures , while increrxsina its versatility and 
respon-si veness to the needs of the occasion, also increas.es the- difficulty 
in "evaluating" those procedures aecording to fixed criteria on quality of 
outcome; By definition, a'Vigid enti.ty is more easily evaluated than a 
flexible entity. ' r: 

9^- vi-i . . 
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It is possible that in the future soM policy makers v/ill deplore the ex- 
isting options for bypassing large segments of fomal analysis in ETAM. 
The existing- exits can be blocked off readily enough— but the choice should 
lie with the using agency. 

VALIDATION OF THE LOGICAL STRUCTURE OF ETAM 

Formal presentations of the procedural structure of ETAM Tasks 1 through 8 
have been given to representatives of t'he sponsor and to associated Navy 
agencies. Although constructive cormients were made that led to changes in 
procedure and nomenclature, no fundamental flaws were revealed in the course 
of these reviews. This is supportive rather than conclusive evidence that 
flaws do not exist, but they do not seem evident to inspection by experts 
,in training, systems and in decision systems. 

SCENARIO APPLICATIONS OF THE PROCEDURE 

ETAM^^procedures were conceived and formalized "around a number of hypotheti- 
cal "scenarios." Several of these exist in preliirilnary working papers, and 
were one major basis for 'cominunicati ons with the project consultant. Dr. 
Gagne. 

Section V of this report contains a full-dress evaluation scenario of a 
hypothetical innovation that- was carried through all tasks up to and in- 
cluding the presentation of decision-making content. This example is the 
most comprehensive "validation" of the ET/\,M procedure to this date. Each 
step could be readily performed. Where the assessor was uncertain about 
expected outcomes, this uncertainty, and its approximate level, could be 
expressed formally. In this sense, a major objective of ETAM to get as 
much information as possible that was available to the assessing team, in- 
cluding the expression of tentativity, was clearly realized. One does not 
need to get stuck and be unable to proceed in the procedures because one 
lacks absolute data at any given point, I'lnurcnce and undertainty, by being 
expressed, become legitimate evaluation data. 

A second scenario treats only a portion of the ETAM procedures, focusing 
more on the selection of relevant variables and on the effects- of decision- 
making upon the management and measurements systems. It was not so much a 
validation of the usability of the procedures as much as a vehicle for de- 
veloping an awareness by decision makers of the potential implications of 
their decisions. v 

A third major scenario run-through was made on a problem assigned, by a 
•representative of TAEG: .shipboard training using micro-image in the form 
of microfiche and standard viewer. The primary objective in this exercise 
was to test the usabilfty of procedural Tasks 1 , 3 and especially 5. The 
results of that- exercise are also reportad in Section V of this report. 
The scenario may be useful in sho>j^ing the adapatability of the framework 
to short cuts, key issues, and especially to the classif icatory structure 
for determining limits to the range of applicability of the stipulated pro- 
posal. If functions are added to the stipula^--.H! innovotion, the benefits 
conferred by these functions in terms of extended range, of training effect • 
can be readily identified. 
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A cost and business assessment was not required of the microfiche assess- 
ment exercise, although the data leading to that phase of assessment were 
pointed to. 

Since in all three of these scenarious the assessors were by necessity also 
the developers of the procedures ^r.their representativeness of future ETAM 
users may be questioned. But no claim is made that assessment based on 
limited information applied to a wide range of semi-knowns will be simple 
or automatic. The written assessments should be evidence, however, that 
the procedure can be made to work. The results of the microfiche assessment 
and the "Three-D Procedure TraineH' assessment revealed that the structure 
of the procedure fostered both creative and analytic thought, and offered a, 
clear pathway through a potential jungle of possibilities. 

ETAM should be validated further by consul tant . level persons- ,.and decision 
makers within the naval training function all of whom have accepted the 
stipulations of training in ETAM concepts and procedures, but who were not 
part of the development team. 

This should lead to the determination of the levels of decision-making 
sophistication that are practical to use for the ^officials responsible to 
make them. But because ETAM is not tied exclusively to any one format of 
presentation to the decision maker, he can elect to use what is congenial 
to him. 

EVALUATIONS OF THE PROJECT CONSULTANT, DR. R. M. GAGNE 

Dr. Gagne is an international authority on education and training technolo- 
gies both as a researcher and as an application development consultant. He 
has shared a continuing interest in pragmatic taxonomies especially in edu- 
cational content and process. He has been intimately exposed to the con- 
ceptual anatomy of ETAM during all stages of its development. His counsel 
has been an invaluable contribution. 

He concluded his detailed and exhaustive review of the content of the Pre- 
liminary ETAM report with the following overview comments in his letter of 
June 11 , 1975. 

On Section V (scenario examples): 

"Seeing the procedure followed in its entirety is highly illuminating." 

On the procedures and report overall: 

"My thoughts about the analysis and procedure are these: 

"1. It seems to me that thjs system proposed is definitely a feasible 
one. Innovations and other proposed changes can be evaluated in this 
way. 
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"2. The procedure ^ascribed 1s v.chly cc;,-,prenans1 ve and thorough. 
It takes Into account what seeins co on trse full range of factors which 
might influence a dacision, a-v: vhich ;„;.-•:•.;: :ci:LriLH:tG to such a de- 
cision. It e;',c.:iTipasses a vcry grsiit range of iiiacarials, hardware and 
procedures that might be proposed for t.ie purpose of accomplishing im- 
provements In tiie cost-effect- vfiness ret io applicable to training opera- 
tions. It considers the cntire/ranye of Navy jobs or job functions for 
which training might be given or proposed to t-e given, 

"3. As a consequence of this comprehan5,1vanf;3S ^ it is a procedure which 
might be described as elaborate. 1 as-,u;ne cnat a thorough procedure is 
what the Navy wishes to have. However, it 'needs to be said that sim- 
pler procedures, involving fewer steps, c??. reci^lily be conceived. What 
any of these would do would be to "cut corners," that is, to make as- 
sumptions which are based upon infontied judgment, rather than upon de- 
tailed consideration of all the sourcos of "hard" data that can be made 
available. Of course, cutting corners also means approaching more closely 
the procedures that already currently exist. 

"4. Surely one of the most valuable contributions of the proposed sys- 
tem Is its approach to the task of astin atlng range of effect. It 
would appear that there has been no" truTy sys tame, tic' v^iyloTo this up 
to now, among Navy procedures. The ideas of Horiving decisions from 
(a) the categories of training outciKas (b) cne levels of training, 
and (c) the effects on job perfonnance repre.'.ent techniques of great' 
originality and also of practicality. Fxist no pv-ocadiires tend to 
neglect these factors, so that tisv do nd'c enter into the estimates of 
effects of innovations in systeinacic ways. A'^ result, Inadequate 
estimations of the effectiveness portion of the ccit-effecti veness 
equation are likely tc be made, Ti,e inciuslon of range of effect esti- If 
mation procedures makes possible tha Gilini latlon .of this potential 
Inadequacy. 

"5. The additional value of ra!ige-o;-a:t.H:i sirjcudures should also be 
mentioned. An Introductory statement in this report speaks of this 
procedure as follows (p. 1-3): "a r-^scrii ci vc a.vi analytic terminology 
that can be applied to data basa t'ejcriptionr '.f ir/:. uructional vehicles 
and their relevant properties, job ...-ius anti :-.nrir relevant properties 
and training courses' and content wl'crvin tr^jUr a i co'virsas." The broad 
applicability of such a planning tool ra-icae^ for beyond the s^'ngle 
activity of estimating cost-effectivMriv^js v' an innovation. 

"6. The procedure seems no less .■lihorough in it; identification and 
estimation of cost factors. Although f:Ji'. q- ■; i ..i" Ir.e cost-effective- 
ness has always sean^ rel-itively the. c -s;c> (>?•.;«•:•..? of the assumed 
existence of "hard" tiara ) , it ii riotablo that the .ources of costs in 
this systsrit appear to be fu]ly covered, whetrurr they seem initially 
large or small. This aspect of the <>yst>m a'^o contributes in no 
small measure to its precision. 

"7. On thp uhc'f, ! cnny ld'. - th- :: r-- - - -^v , -^o wliich I have 
cofitributed only uS a critic «.;c ..•>,<,.". ua-.;o: , a vofy substantial ac- • 
complishmant, of outstanding Ciiccsuui i soufr^nus,-. ." 
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This ends the quotations from Dr. Gagne*s evaluations. His point 
number 3, directed at the preliminary version of ETAM, has resulted 
in changes in the final version which emphasize and underscore oppor- 
tunities for "short-cuts" and procedural simplifications where the data 
available and the nature of the decision justify short-cuts. 

Dr. Gagne, commenting on recommendations in the report for further 
ETAM effort, endorsed the studies proposed, and made the following 
assertion relevant to evaluation of ETAM: 

"Developing a computerized assessment model ... would indeed be a 
challenging task. It would seem that ETAM in its fully conceived form 
could only be evaluated in this way. It would be good to see it done." 

COMMENTS ON THE DECISION MODELS 

The primary validation emphasis was on the use of the decision- tree frame- 
work outlined^in Tasks 3, 4 and 6. One validation question was concerned 
with the structure of the -decision tree itself. A second dealt with the 
approach used in handling utility functions with multiple attributes. 

As a result of the concern that the original decision tree did not permit 
variations in the way risk reduction projects were packaged (and thus did 
not_ permit combinations that might perhaps be more cost-beneficial), the. 
decision tree was redesigned to allow the decision variable to be calculated 
for more than one project grouping. Also, the new tr.ee incorporated a 
project success probability not available in the original tree. The new 
format resulted from a review of the decision tree framework by Dr. William 
Giauque of the Naval Postgraduate School, Monterey, California. 

A second key question concerned the use of the additive (rather than the 
multiplicative) model for multi-attributed uti lities . Dr. Giauque was 
again consulted: A number of cautionary- notes were inserted into the pro- 
cedures based upon his recommendations, however, it appeared from formu- 
lating several analytic possibilities that the additive form would be valid 
for the high majority of proposals with which ETAM wouMbe concerned. 

It is recognized that the scaling methodology for the "importance" factor 
needs further study. and refinement. An ETAM predevelopment study has 
been recoranended in Section I of this report. 

CONCLUDING REMARKS 

The logical structure of ETAM as a conceptual model of assessment param- 
eters that are comprehensive and relevant, has not been challenged. These 
include the taxonomic parameters which lead to (a) identification of key 
rejection factors in assessing an innovation (Task 3); (b) potential range 
of effect analysis (Task 5); (c) presentation of assessment data to the 
decision-making process (TasRr 4, 6, 7 and' 8). 
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The center of concern has been the Implied amount of labor and level of skill 
for responding to the apparently large number of steps and alternatives (as 
offereQ by the cost and benefit uaxonomiiS; ccnuaincd in the procedures. 
Figure VI-1 shows rough estimates^ of time to apply ETAM to a range of innova- 
tions. The bulk of the voluminous report itself is, intimidating. The ques- 
tion is asked: "Is the outcome of 'an assessment worth all this trouble?" 
This is a cost-effectiveness question, and can be answered only by refer- 
ence to the costs and effectiveness measures of existing or alternative 
practices in reaching decisions about acceptance or rejection of innovations 
or other structural changes in the Navy's education and training picture. A 
clear-cut "decision point" often cannot be ascertained in traditional prac- 
tices but rather what appears is a network of localized pressure points, and 
unshared rationales and objectives. Neither the costs^ nor the benefits, 
liabilities, and risks can ^be determined from such processes. 

Revisions in the ETAM statements of procedure in the preliminary version 
have emphasized where and how short-cuts can be taken. In this respect, at 
least opportunities for "simplification" have been offered and justified. 

But this still leaves the large bulk of paqes making up the procedural "map" 
of ETAM, a formidable tome comparable to the size of an operating and mainte- 
nance manual for a piece of radar gear. It is proposed that ETAM be imple- 
mented on a conversationally interactive display tenninal. This facility 
will simplify the 'procedure in several ways. One, it suppresses procedural 
information that is not pertinent to the analytic .^out3--and stage, in that 
route--which is of significance to the assessor. The user retrieves selec- ' 
tively both type of content and level of content. Two, it relieves the 
user from the labors of cornputation* Three, not only are appropriate data 
formats presented for the entering of data values, but these data values 
are automatically transcribed to output formats in vvhatever variety have 
been established. The computerized implementation would enable a "human 
factors" validation, as contrasted with the ''logical validation" of the 
ETAM procedures.. 

The reduction of clerical paper shuffling and arithmetic will not, however, 
eliminate "complexity" from the task of making predictive assessments. This 
complexity is inherent in coping with uncertainly ;na incomplete information 
in the problem to be solved. Finally, human judgments, of "relative impor- 
tance" and "probability of given outcomes" will have to be made, however 
simplified the format structures for making tliese judciments. In general, 
the more important and far-reaching the outxcme, the greater the range of 
uncertainty because of the incompleteness of "'hurd data." It would be dis- 
honest and dysfunctional to disguise real depondenciJi; on informed human 
judgment or the effort and cost involved by tre .exorcise of that judgment. 
The act of creating useful information where it does not exist is not simple 
even though it has been "simplified." 

BecausI ETAM was explicitly designed as a too"* to serve human users, its 
validation cannot be independent of the skills and diligence of the user in 
the role of assessor and in the role of decision maker. 
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Task 1 
Task 2 
Task 3 
Task 4 
Task 5 
Task 6 
Task 7 
Task 8' 



SIMPLE 



Hr. 



1 

16 
8 

12 
4 
8 



50 'Hrs. 
1.25 Wks 
.29 Mos. 



■TY PE OF INNOVA TION 

AVERAGF COMPLEX 



4 Hrs 



2 
40 
24 
16 

8 

•16 



106 Hrs, 
2.65 Wks 
.61 Mos. 



8 Hrs 



(Not Included) (Not Included) (Not Included) 



160 
40 
24 
12 

.40 



287 Hrs. 
7.175 Wks. 
1.B5 Mos. 



Figure VI-1. ESTIMATES OF TIME TO PERFORM ASSESSMENT USING ETAM PROCEDURES 
. (ASSUMES AVAILABILITY OF AUTOMATED PROCEDURES) 
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This level of validation awaits the future, and would be meaningful not 
so much in the formal exercise as in the nature and scale of adoption 
ETAM represents systematic rationality on a broad scale, and in many en- 
vironments now, would be a cultural as well as a technical innovation 
Cultural acceptance is relatively independent of technical validity if 
indeed the two forms of acceptance are distinct from each other. It should 
be conceded that introducing ETAM into adoption may require inventions for 
cultural acceptance at least as substantial as the technical inventions 
required for its creation. 

Many of the recommendations for follow-on efforts' are directed. towards 
cultural acceptance in the milieu of its intended use. Because it is 
completely explicit as a decision-making tool, its inadequacies can be 
identified and remedied with usage. As a structure for retaining archival 
data, it also enables the quality of decisions to be assessed and thus • 
potentially Improved. Because of its scope, decisions in training and 
education can become coordinated rather than competitive. This would 
be an ultimate validation. 
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APPENDIX A 

SUPPLEMENTARY DESCRIPTION'S OF TASK FORMATS AND TASK ELEMENTS 
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This appendix includes extended descriptions of the task 
structure elements and their training and learning implications 
It is intended to complement the material in Task 5 of the ETAM 
Procedures (Section III). 
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TASK FORMATS IN' HUMAN PERFORMANCE 

Introducti on 

Travelers from the Drifted States" to the United Kingdom, who overcorne their 
initial fears, may be pleasantly surprised to discover how quickly they can 
"reverse" many of their driving habits. .The .British gearshift is operated 
by the left hand rather. than hy the right hand, the driver sits on the right 
side of the car rather than the left, looks to the left for the rear view 
mirror rather than to his right, and drives on the left side of the road 
rather than the right. Many of these reversals have to be performed simul- 
^- taneously such as when making turns at intersections. The need to reverse 
scanning patterns for traffic liabilities still further complicates the 
adaptation. Yet in^a relatively few hours, depending on survival during the 
first few minutes, the driver begins to be comfortable and can even divert 
h-is attention to viewing the countryside. The adaptation is made in perhaps 
a thousandth of one percent of the time rsequired to learn the original driving 
skill. There is, of course, large individual variation in such adaptability, 
but this variation may be largely compounded of attitudes towards making the 
change rather than "information-processing" capabilities. 

If we think of the driver's ski IT only as a set of required fixed linkages 
between what he sees and what his hands and feet do, it is impossible to 
account for this rapid adaptation, although it does help account for some of 
. the errors he will tend to make. In addition to, and partly independent of, 
-these eye-hand linkages he has acquired a model or "format" for receiving and 
prpcessing classes of information that relate (a) his purpose of the moment, 
(b) his pro^'ection of the track of the vehicle into the roadway "corridor" 
ahead, and (c) the control variables of direction (steering), velocity (accel- 
erator and brake) and power ratio (gearshift positions) . The input variables 
and the output variables in driving remain the same^ although the data content 
is very different Wis this processing fonnat which, when established by 
learning a task, can mediate substantial shifts of 'learned patterns of be-, 
havior from one context to another. 

When a^brilliant marketing executive is hired away from a soap company to 
organize a marketing operation for a clothing firm, his knowledge about soap 
and soap markets had better be left behind; what will be usefuTwill be his 
.conceptual models and formats about market development strategy that are at 
. least partly independent of the product. The ability of the experienced pro- 
grarmier to learn new programming languages quickly is at least partly mediated 
by the constants in program language structure and entities. 

The. design of training and the design of task tools and supports caa- be enhanced 
by a workable awareness of how formatting occurs in tiie learning process, and 
the various adaptive functions it supports in the economics of behavior. 

Definition 

— . \ 

A distinction should be made between the des^cri^jon of a task fQmiat and the 
embodiment of a-'task fbrmat in task a-^tivities. Ti." »-n;hodiinent of a task format 
in huiTjan performance consists of situational variab-les and the response 
variables that recur when the task is performed many times. The value assumed 

A- 2 
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by each variable may change on each occasion the task is performed. Thus, in 
driving the car, the picture of th6 projected track or corridor for the auto- 
mobile through traffic and the transient states of velocity and direction are 
never twice the same. But the operational variables as variables, are con- 
stantly present each time the task is perfoy^med. 

The format of filling out a purchase requisition is explicit in terms of the 
information variables printed on the form. The requester of the service 
making -the purchase requested by the requisition is given the standardized 
data names that will yield the information essential to providing the service. 
The printed form is a cultural invention which acknowledges a "task format" in 
the sense described here. Despite complaints, the printed form is one of the 
more powerful labor-saving devices for handling transactions among organiza- 
tions that request and supply services. 

The human is capable of learning a large variety of tasks having a variety of 
structures: perceptual, interpretive, decision-makingj constructing, per- 
ceptual-motor. The variety of tasks is somewhat paralleled by variety in task 
formats: we shall be able to see similarities among tasks that superficially 
appear to 'differ , and these similarities will help account for the human operator, 
having learned one of these tasks, to learn quickly the other members in the 
task croup. It would indeed be useful to know if, perhaps a few dozen task 
format "archetypes" might account for sixty percent to eighty percent of work 
tasks in our culture, ^ • 

The reader is again reminded that^^aTtask format can be thought about with either 
of two different refe-rences in^nffnd. From one point of view, the reference i.s 
to an external structure such as a printed transactional form while from 
the second point of view, a task format is part of the behavioral pattern that 
is learned by an operator in the acquisition of a skill or ability. Both mean- 
ings of task format have utility for the intelligent design of task supports 
and Jtraining, and for expectations of transfer of training by the operator from 
one set of work requirements to another. But unless qualified, the expression 
"task format" in this paper will refer to a learned pattern of organization in 
behavior rather than an external representation of a format. 

Thd Learning of Task* Formats . 

When regularities are repeated in human experience, they become identified and 
anticipated. This process whereby general patterns are abstracted from the 
mass of variations occurring on repeated occasio>^s takes place spontaneously 
in learning the task. Thus, as an individual user of a computer terminal I 
have learned that the following things have to happen when I use the facility: 

' The device must be turned on. 
I must identify myself with an acceptable code. 
I may have to respond to an authorization request. 
I must identify the program context I plan to use. 

I may have to cope with the system denying me service because it is too busy. 
I must request, by suitable identification, work that I have stored in the 
system on previous occasions, or state that I am engaging in new work. 



RIC 



A-3 



4 :7 



TAEG REPORT NO. -.2-3 

Lg some applicati-ons several of these operations may be procedurally merged, 
but I am aware that the variables nevertheless exist. If I am to learn to ' 
sign on and use a new application, or a new terminal -computer configuration,' 
I spontaneous ly ask for the specifics of procedure on each of these variables 
This rormat not only structures my getting information about a modified task 

content;, it also helps me remember it and apply what I have learned The 
model may also help me interpret why something goes wrong when it does Even 
in a new set of circumstances, I can generate a set of "diagnostics" for 
probing into a difficulty on the basis of this conceptual structure of vari- 
ables in this class of activities. Within limits, this format enables me more 
or less readily to substitute one response mode for another: pointing at menu 
selections with a light pen or entering the data from a keyboard, or calling it 
over a telephone. ,„ j > > a >- 

The greater the regularities in the presence of the' same variables from one 
task occasion to the next, the more rapidly will implicit format develop 
Regularities consist in: the same set of variables; the same sequence in time; 
the same arrangement in spatial location. If the sequence in which the operator 
must cope with. a set of task variables changes from one occasion to the next - 
that IS, the sequence is variable or random - the task format will be learned 
and be a part of performance somewhat more slowly. That is, more repetition of 
learning trials wi 1 be -necessary. The same will be true of spatial position 
of the task variables in the field of regard or in the work environment. Where 
.there is standardization both of sequence in the operator's coping with given 
task variables, and in their absolute or relative soatial location, the 
redundancy increases the rate at which the task format becomes learned as a 
factor in performance proficiency. It is a fundamental principle of learning 
ihf .Jf ^ '5'°^'"* of diminishing returns, redundancy of task cues increases 
therate of learning and favors retention and reinstatement of the skill over 
periods of disuse. The principle is significant, even though in highly practiced 
performance the operator gradually tends to dis^^eyard redundant cues by selecting 
(through spontaneous processes) a cue element that is sufficient for selectina a 
task response. . ^ 

To the extent that, in early learning of a task, the learner engages in activ- 
ities during practice that are irrelevant or extraneous to the prffcessinq 
format to be developed, the format will be learned slowly. For example, tro'uble- 
shooting electronic equipment can be divided into two major classes of activity 
(Ij the procedures of setting up test equipment, applying probes, taking measure- 

??rur;inT''';'?,^'^K' ^^^^^'"9^' ^'^''^^ing in manuals of ?n - 

?n mSkp Ihfl •? *^?^3pplication of strategy in the selection of that next test 
to make that will yield the largest amount of diagnostic information, and the 
logical operations of deducing the significance of a given test fi^nding as to 
what sections of the system may be excluded from fu'rther suspicion. Where 
practice exercises in early stages of learning mix these classes of activities 
the formation of strategic and logical skills may be very slow because' of the ' 
confounding and attenuating information required for the procedural operations 
Obviously, the exercise of the strategy in the real job .demands integration of* 
the activities, but the formation of the format structure may require at least 
temporary separation during training of these -two sets of activity. The analysis 
of the psychological functions of task formats in a later s.-ctiop will reveal 
the justification for this separation. (It is possible that a task format can 
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bg so overl earned in artificial conditions, that it may not adapt to combining 
with other formats which may need to be exercised and integrated with the 
first: stage of learning is an important dimension for the design of efficient 
training schedules.) 

The "Description of Formats" and "Behavioral Formats" < 

It i's^ possible to memorize the bidding rules of the game of bridge as a set of 
verbal instructions. If they can be recalled, these verbalized rules may ^ 
jissist in making appropriate bids , but in themselves they are insufficient for 
excellence in bidding even if the psychology of bidding were to be set aside. 
When the rules are embedded in the skill of sizing up a hand, the bidder can 
cope with exceptions to the formal rules, making tradeoffs and treating the 
elements in his hand as a pattern. He may add to the mere bid count in his hand 
an imaginary playing out of his hand against his opponents, and add the imagined 
outcome to his assessment of the bid value. The verbalized description of card 
value count making up., a bid serves as an external guide that should help him to 
learn the format as a skill component. In the sense that it is a guide to 
behavior, it is of course a format by definition. 

Take the example of an expense account form. The variables on the form specify 
the kinds of information that ordinarily are necessary • and sufficient for sub- 
mitting a statement of trip expenses according to the' requi rements for accounting 
and reimbursement. It is the external format for generating a statemejit of trip 
expenses. It may be contrasted with submitting a statement in the form of an 
essay. The printed format usually helps the person ^eafn comparatively quickly 
a psychological or behavioral format for selecting data from perhaps a r^andom 
^ assortment. of notes on expenses, and to quickly translate them into entries on 
the paper. The implicit format for expense accounting in the employee's head 
will also tend to guide him during his trip as to what expenses he makes nota- 
tions of, collects receipts for, remembers and qualifies.. The accounting 
format, when in his head, guides his spending or at least his attention while 
spending. The format becomes operationally meaningful beyond its printed 
embodiment. ' . 

ihe point here is that it is naive to assume, that because a learner has been 
told or shown a format structure, it will thenceforth serve as an effective 
component in his skill. Formats become learned and effective in skilled per- 
formance as one learns to ride a ^ bicycle: by repeated practice. 

The Behavioral Value of Learned Formats 

Recall that, in general terms, a task format is a framework of those factors and 
variables in information and response that tend to be constant in many repeated 
cycles of the task. When a format becomes learned as a component structure in 
a task skill, it serves as a behavior organizer in the following ways. 

1. Directing attention . The task format embodies selective attention. The 
operator learns what variables or properties of the work environment are' task- 
relevant, and those that are irrelevant to any task action. When the operator 
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has learned what specifically to look for as essential work cues, and can 
disregard the non-essential, and irrelevant, the burden on his attention and 
information processing filters is substantially .^educed. It may again be 
emphrsized that the format directs attention to catego ries of task cues, 
''stimulus variables" or "task parameters" — where eacFof these expressions 
IS intended to. mean about the same thing". The effect of the format is, there- 
fore, both that of a selective filter and that of a scanner for what have been 
learned as "essential" categories of task data. When attention becomes auto- 
matically directed and selective, the further processing of task data can be- 
come simplified. • 

« 

Efficiency is, however, bought at a price and the<.price usually is some degree 
ot rigidity. The task format can be practiced by repetitive experience to -the' 
point that the operator disregards — fails .to notice--- what should be 
qualifying conditions, or out-of-the-ordinary demands. His perceptions can be- 
come stereotyped so that he operates on the minimum standard format. His per- ' 
formance now becomes vulnerable to a demand' for behavior that requires accepting 
and processing information that is outside his format structure.* . ' f 

The determination of what are the "essential "' cue variables in a task does not 
arise by magic from some abstract specification of ideal .task performance. '-It 
arises from the psychological operations by which input states, goal concepts 
and output actions are linked to consequences as perceived by the le.arner over 
a substantial number of learning and performance trials. (All performance has 
some learning effect.) By the general principle of least effort, continued 
practice will result in the processing of the least amount of information that 
is sufficient to produce results that are adequate or "satisfy" the performer -- • 
•or do not interfere with his satisfactions, which is perhaps a di fferent 'picture 
Understanding this principle should help in the progranr.ing of learninq 
experiences. . ■ . . 

There is irony in the development of perceptual rigidity. As the learner acquires 
perceptual formats, the demands on his attention dealing with standard task data 
tend to become less, so that in effect he can disperse attention over a larger 
field of situational content. He, therefore, is acquiring the capacity for more e» 
flexible pey^-ception, both in range and detail, but he must be incfeTtb exercise 
this channel capacity. If he must learn to perform several streams of task 
activities more or le'ss concurrently, this is in part how this added channel ' ' 
capacity is acquired. 

It is possible for the enlightened learner to continue to impose on himself 
continuously more rigorous criteria of attention to nuances in input information 
after he has mastered the ability to process the minimum required data in a 
task -- that is, he uses the task format as a performance tool rather than being 
used by it as a tyrannical constraint and limit. This assertion assumes some 
cognitive freedom of choice rather than the notion that learning must be acquired 
only through external "shaping" of behavior at the time of skill acquisition 
There are, no-doubt, large individual differences, perhaps arising from many 
causes, in this self-instructing phenomenon. 
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In conclusion, the learning' of 'a task format has a powerful effect: in restricting 
the r^nge of what the .operator pays at;£ent*^on to and notices in the "total 
stimulus environment of the^task. Thjfe restriction and , focusing of .attention 
is one factor in the. improvement of qua^litative and quantitative factors in, task 
performanc&>,.^The greater the standardization of information-categories, ap'd the 
temporal- and spa1:i al 'presentation, of given categories of " task information, the 
more-jj quickly Will the format'be learned as manifest in selective attention. If-, 
the task demands sainp-ling for g^riodic changes in value df given task input 
variables^ the standardization tff spatial locatioji of the indicator for the 
variable assumes spepial significance When the^operator is highly loaded or 
5 tressed. * * " ■> 

2. Organizing short term ^Riemo ry . Short'term memory r- or "working memory" — 
is the*- retention of variable information peculiar to a given cycle of task per- 
formance. It ranges from keeping in mind one's "p^^l^ce" in a procedural sequence,, 
or perhaps briefing instructions for a parti cular^^t of job operations, to the 
many factors of demand and resource and sequence in formulating the solution to 
a complex/logistics problem. For practical purposes, the complexity of the 
problem which the human can solve is limited by the amount of informati9n he can 
effectively hold and use in his short term memory. Sport term memory can, of 
course, be assisted. by job aids which, like a displayed map, may refresh the 
content in short term memory but cannot ^erve as its equivalent. 

At any given moment in a work cycle short term memory tends to hold representa- 
tions of task conditions and status, momentary purpose, and perhaps a symbolic 
key to the invoking of a class of response operations to be performed on the data 
Whereythe operator's experience has encountered a high degree of regularity in 
ttme; space and data content of task stimulus and task response, the content of 
short term memory tends to become reduced -to merely that situational data suffi- 
cient to evoke the next response of execution* To the extent that a stimulus 
sequence is invariant, the response sequence may be invoked and executed without 
attention to the elements within the stimulus sequence, but as an automatic 
train of response. * - ^ 

Since performance is limited by the ability of short tenn memory to cope with 
task information, both the design of the task and training for the task should, 
be directed towards the striifturing of short term memory content. This structure 
is based upon the task variables which make up the situational context to li^htch 
task responses are made. . • 

With practice on a task, the learning processes seem to sor-u cut thp recurring 
task variables sa that short term memory develops the equivalent of a matrix 
or template for the reception and implicit "labeling" of classes of task data. 
A limited (and partially misleading) analogy is that of settihg up a set of 
standard pigeon-hole locations for task data variabiles. The analogy breaks 
down because there is interaction among the contents of pigeon-holes: pattern 
as well as elements may be perceived by the operator the content may operate^ 
dynamically rather than passively like impulses written into the retgister 
positions of a computer. Thus, the driver of the car may implicitly note his^-. 
own speed and direction and the speed and direction qf another car, and more 
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less continuously compute the projections of the data samples with respect 
. to collision probabilities. Or the symptoms reported by the patient to the 
diagnostician may con-tinuously sumrnate i.i>^ihe physician's, mind into a pattern 
which he recognizes as a disease entity. . . - 

The value to performance" of this parameterizing of task data is clearly 
enormous. That task data which demands differential task response is sorted 
■put from that v/hich is either irrelevant to the moment, or calls for a 
'•standardized subroutine or pre-established responses". The psychological 
categorization of data enables it to be apprehended and stored both as data 
elements in the matrix, and as a pattern to be completed. In an operational 
sense, short term memory has becom^ "organized."' Increased practice "on the" 
task increases the stability of this organisation of input^ata structuring. 
The result is an effective or functional increase in the capacity of short < 
term memory. , That is, to the eijctent that existing data patterns are well- 
structured in short term memory, additional data can be stored in it. The - 
driver with increasing competence can note and remember additional data in the 
driving environment; the diagnostician can hold In mind a larger set nf data 
specific to this patient and his ailment; the architect designer can optimize 
a larger set of design parameters and constraints in the course of. design. 

A major function for task performance sefved by short te>V memory is the 
capacity to sepa(;;,ate stimulus information temporal ly from response execution. 
Thus, the accomplished typist apprehends a group of letters and words as a 
clump which is stored in. short term memory which is later executed as a clump 
by her fingers on the keyboard. A contrasting, organization of work would consist 
of looking at an input character in the source document and keying that, character, 
then looking at the next. character and keying it. The temporal separation of in- 
put from output enables the operator to prepare adjustments to momentary complexi- 
ties of response demand without a disturbance to the pace of the output. Vari- 
ations both in the complexity in the input and complexity in output response 
tend to be smoothed out by some "separation of rate in the reception of stimulus 
input and execution of output. A pattern of input is apprehended and briefl'y . 
stored and matched to a pattern of output. All of these factors together reduce 
fatigue in the performance of the task, and delay the onset of "blocks" -- a 
penu^d of time when within the operator, presumably from fatigue, there ,is a 
psychological block between input processing and output processing. These are 
also occasions for errors. 

To the' extent that the rate of apprehending stimulus input is faster than the 
rate of response execution, the operator can attend, at least intennittent-iy, 
to otner streams of information. For example, he may monitor the outcomes of 
executed responses." ' 

Short term memory mechanisms can be 'viewed, therefore, as information buffers 
that enable temporal flexibility between the presentation of a task stimulus 
and its action in eliciting a task response. This flexibility exists as a 
capacity. How this capacity becomes an operationa-1 ability depends largely on 
the conditions and extent of task practice from which no skill can be inde- 
pendent. The operator Ifearns what he does. Wha^: he does may be modified both 
by external circumstances and by what he tries to do by voluntary processes. 
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The buffer in short* tern', maa^ory may be tnought of as consisting of a set of 
register^. Each register position has become specialized to receive the data 
value of a -given task variable. When ihe . registers in tlie buffer are filled, 
the content as a pattern is dumped into an action converter. This mechanistic 
description is intended only as a highly oversimplified analogy ^nd should in 
no way be treated literally. The point to be understoojiJ-S-'4+iVf ateiit^^^ for 
increased processing efficiency when processes of this kind result in the 
organiza'tion of short term memory content. 

3. Task closure and reset . The learned structures that operate iri' short term; 
memory may also y.ield the phenomenon of the sense of task completion. This 
source of completion may be accompanied by the erasure of much of the variable 
content in short term memory. Thus, one looks up a telephone number, and per- 
haps keeps it in mind until the digits have been dialed. On the completion of 
the dialing, the number may tend to be forgotten at once. The "space" in short' 
term memory is now available for other semi-transient data. And the human has 
an accompanying sense of completion: the task or sub-task is ''closed". 

What constitutes a task closure depends on how the t^isk is learned. When a 
telephori'e operator mediates between a caller and the station or person being 
called, her task is ordinarily not completed until she verifies the calling 
number. She wiTl, therefore, tend to retain the calling number in mind (assuming 
it hasn't been written down) until it has been verified. When it has, then and 
not until then, will she have closure and the unbuffering of the transient 
content. The understanding of this phenomenon should help in the design of task 
training. Thus, if a pilot or a maintenance mechanic has extensively practiced 
and learned his primary tasks as being completed without fil ling in a debriefing 
report, he learns closure before filing the resport of what happened during the 
task. If now he must file the report, his memory of task data may be unreliable 
and incomplete. But if filing a terminal report is learned as a normal portion 
of a work cycle, his closure will be delayed until completion of the report. 

These comments must be qualified. Closure, as we know from everyday experience, 
is not all-or-none. Insofar as forgetting is concerned, it is a matter of 
degree. Since tasks are composed of subtasks and also of higher order tasks, 
closures are also hierarchic. Furthermore, there are relationships betwee^n 
short term memory and long term memory whereby some content in the former be- 
comes content in the latter. But mechanisms of relationship between short term " 
memory and long term memory are outside the scope of this paper. Task ^'ormats 
are retained in long term memory from which they are accessed and enter opera- 
tionally into short term memory under cues that initiate a given task activity. 
A^person is "set" for a task or a task cycle when the task format has entered 
short term memory from long term memory. 

A final practical coimient. The learning and performance situation of the learner 
can be such as to foster the sense of closure when he has completed some ter- 
minating task response. If so, he will tend not to search for evidence of out- 
come adequacy of his performance, nor to link efficiently (in the psychological 
sense) such outcomes with the elements of task situation and task response. 
This pattern of behavior will tend to preclude the progressive improvement of 
his own performance with respect to given criteria except by externally imposed 
instruction. The most effective context for learning occurs when the information 
of the task cycle is still in mind and the resulting outcome is presented while 
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that information is still "live" in present memory, >/hich must be preliminary 
to closure as described a^ove. This observation suggest\not merely a 
temporal connection betv.een performance of task and presentation of outcome: 
it also points to what the operator has learned as the completion status of a 
task ^.ycle. These factors suggest why so much academic study aimed at ''fiaish- 
ing the test" is not only inefficient but poorly structured for enabling the 
student to become an effective self- teacher> 

4. Transfer of skill . The task format, like a template for inserting the 
values of variables in a more or less standard equation, enables large vari- 
ations in the values of the situation variables to be processed effectively by 
the operator. The format may thus play a major role in enabling' rather sub- 
stantial changes in task content and task setting to be learned quickly, 
relative^to a control condition where the new tasK was learned without benefit 
of existing format. 

Perhaps it should be underscored that a higHly routinized task that is always 
performed in an invariant way to an invariant sequence of readily identified 
stimulus elements may, when practictid many times, be performed purely as a 
rote sequency with a minimum development of a transferable format or abstract 
or implicit model of the task. Formats develop to cope with variability within 
the task and from one task occasion to the next. ' In other words, where the task 
"variables" assume only a single value, and little or no information is carried 
over from previous steps to later steps in a sequence of actions, the procedure* 
will tend to become automatized. Little transfer is to be expected, unless the 
operator learns a variety of tasks having the same structure. Lp this case, 
the conditions of variability are met, and a task format will tend to develop. 
The result will aid progressively 1n still more new tasks having simj.lari ties 
in structure. For example, a shop man learns and practices for years taking 
the same kind of bicycle apart and putting it together": He will be inept in 
learning to disassemble and assemble mechanical objects different from that 
bicycle say, a gasoline engine. But a person who has takpn apart and put 
together a variety of objects learns a general pattern, including some strat- 
egies^ for this class of work. He can, with comparatively littVe guidance, 
disassemble and assemble devices quite strange to him. He has, acquired a 
general format for disassembling and assembling devices. 

II 

T|iere are, of course, factors other than task formats which can be responsible 
f.or transfer of training: a comnion base of nomenclature and reference informa- 
tion between the old and new task are examples. 

. Types of Task^ Format ■ " . 

Task formats are as nunserous as there are classes of tasks. This writer, in 
several reports (1969, 1973), has analyzed several dozen that might be regarded 
as widely distributed in the domain of cognitive and perceptual -motor work. 
Ih^re are a few which may be regarded as archetypical. 

Verbalized self-in structions and external fbrma ts. Task performance can be 
structured by verbal instructions given to oneself or given by a manual or an 
instructor. The, information in the verbalized instruction may contain informa- 
tion necessary for actual performance, but not suff'icient. There must be a 



TAEG ktPORT NO. 12-3 



t 

capability for translating the verbal instruction into perceptual actions and 
muscular or other output actions. The self-instruction is an e xternal format 
for performing the task: it guides bciavior without in itself "Being that 
behavior. 

A printed form to be followed in a transaction is another example of an 
external format. It specifies the classes (variables) of information necessary 
to obtain in order that a given action can be taken. The printed form not only 
facilitates the performance of the task of getting and setnna down the essential 
data.it also simplifies the learning of that task as contrasted with a situation 
in which such a guide is absent. It enables the operator to restrict his opera- 
tions in getting the data content: he does not have to think about what kinds 
of data to provide. (Obviously, efficiency here has its tradeoff in flexibility 
and adapti veness, but we cannot address all issues at oncel) 

There may be procedure formats. An example may be that of formulating a search 
specification into a data base system. The operator may have the task of 
searching a large data base of electric motors and parts by attributes and 
functions associated with given parts. This reouires him to name attributes 
and the associated attribute value as well as to indicate logical connectors 
such as AND, OR or NOT between elements of the search request. A graphic dis- 
play may show him boxes and box names to fill in -- a standard format for 
creating the specification of his inquiry; he may pr may not have to follow a 
prescribed sequence for entering the data. This is an external task format. 
He may have to learn the format -- as contrasted to using a "free form" of 
inquiry specification — but once the format is learned, he will tend sponta- 
neously to think of his request in this format. The' format becomes internalized. 
When it is, it will take less of his channel capacity (and probably produce 
fewer unintentional errors) ttian if he were to continue in "free form". But 
even allowing him free form mode would, with repetitive experiences, result in 
his developing (eventually) his internal format which standardizes the opera- 
tion for him. Repeated human experiences lead to regularities in behavior 
which is, of course,- a major characterization of what happens in learning a 



Procedural sequentialities . If an individual is placed in a variety of situa- 
tions all of which have a general pattern, he will tend to learn and anticipate : 
the elements in the pattern. Thus, the operation of a mechanical vehicle 
generally requires getting into it, turning it on, engaging power' to the 
wheels, releasing a braking mechanism and preceding. Most of these operations 
apply equally to an automobile, aircraft or boat. Depending perhaps partly on 
intelligence, but more on the variety of experiences, this pattern of sequen- 
tial ity can be adapted to novel situations. 

" '■> 

Operations with a weapon have tlie general sequence of preparation for use, load- 
ing, aiming and firing. It is not necessary that the individual be able to 
verbalize the format in order to have it expressed in his behavior. The fcrmat 
is expressed ir\ behavior by the relative rate at which he can lea>^n the task 
as compared witjh a control subject who lacks the experiencas for having developed 
the format. Clearly, a format in behavior may have many individual, levels of 
specificity and generality. No sets of haoits are alike in any two individuals, 
andan internal format is a learned entity that, in psychological jargon, is a 
habit. 
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Transactional formats. The general structure of a transactional format consists 
of the following constituents: (a) a set of more or less standardized input 
variables to a processing function; (b) a set of variables of reference informa- 
tion that supports the input data for the processing function; (c) a set of 
more or less standardized output variables from the processing function; and 
(d) the processing function itself. Some transaction structures may include a 
comparator oi the processing output with an idealized or criterion output and 
n the actual output is out of ^^rescribed tolerance, a feedback arrangement 
leading to a recycle of the input with the benefit of the added corrective data 
In the case of interest to us here, the processor is a person. 

An illustration will help to tpranslate this definition into a clearer picture. 
A dispatcher takes trouble calls and selects what repairmen will take what calls 
and in what Order. His i nqui ry .about the trouble will tend to consist (in this 
particular task) of a pattern of the same variables: nam and address of the 
caller and geographical location of the' trouble, the urgency of the trouble and ' 
the nature of the trouble. He has learned to extract those essential classes 
OT data zhaz nelp him decide (a) the priority to assign to the service if he 
'M\ °J requests; (b) whom to send; (c) what supplies and tools to send; 

\d) whether to send only one or more than one man. These are the output vari- 
ables in his decision-making. His reference information may consist of the 
following: (a) the implications of any in a given set of reported troubles to 
the customer; (b) the implications of a given class of trouble for the company; 
[c) the relative importance of a given customer; (d) the -abi 1 1 ties of each of 
..IS service crew; (e) the availability of members of his service crew with 
ho^Slon i^^^ who could respond to the trouble call. He attempts a best match 
Detween the variables in the -demand for service and the properties of the re- 
sources available for responding to the demand. His mental processes leading • 
to the decision may include various strategies and tradeoff values. 

This describes decision structure. (It also is a specification for a procedure 
to devellop data base conte-.L and structure for the support of human decision- " 
crhIZ-" ^Ht^ variables, more or less, apply to any dispatching and 

scheduling task. The task format that is learned by this dispatcher may be 
transferred, with various degrees of savinas in learning, to a wi^e variety of 
contexts from .telephone repair to the handling of contingencies on the shop 
floor. (The greater the overlap of reference knowledge variables between the 
old and new tasks, the greater the additional savings in training.) Obviously 
the extent to which the new task to be learned has explicit parallels in its 
structure of variables to the old task, the more rapidjy will the learner adapt. 

This transaction structure is also applicable to the understanding of task 
formats dealing with multiple-variable tracking tasks such as operating 
vehicles. This form of analysis is, of course, precisely the approach that the 
engineer and computer prograrmier must take when tPey try to automate a manual 
p ni?-^i I ^[1^ identify the input variables and the control variables 

Some external arrangements of the task make implicit formats easier to learn 
and reinstate, and generally offer more flexibility in performance. For 
example, if the entire set of task variables comprising the format is displayed 
to the operator at one. time, like a map, the patterri of variables will more 
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quickly be learned. The example is tnat of the printed expense account form, 
or the engineering change fo.rm. The alternative arrangement, fostered by 
limitations in computer display technology, is to display only one variable 
at a time until the operator responds to it,-- the remainder of the trans- 
action is hjdden from his view. The same principle of showing the entire 
structure all at once, rather than a fragment at a time, is probably' desirable 
for training, perhaps even where the operational environment may present only 
one or a few variables at a time. 

V 

Obviously the visual layout of the external format should parallel the sequence 
in which information is received, and is grouped and organized for reaching 
decisions. If reference information is externally provided (that is, external 
to the operator's head such as a display screen) its layout in terms of data 
names and attributes of entities should be consistent with the implied demands 
of input variables, output variables and processing strategy in the decision- 
making operation. 

Even though a learned task format can mediate transfer of training across sub- 
stantial variations in task setting and task content, the larger the number of 
identities between stimulus layout and response selection, the more rapidly 
can the operator learn the second task. 

Psycholinquistic Formats 

In no existing field of behavioral inquiry has the construct of "task format" 
been so widely investigated as in the psychology of grammar and psycholiguistics 
the study of structure in verbal expression. It is indeed a fact that the 
"rules" of grammatical constructions become embedded in our verbal behavior, 
although the rules may be manifest in well-nigh infinite variation of content. 
The structure of "format" of the sentence as a "unit of thought" seems funda- 
mental, as are the sentence structure variables of subject, predicate and 
object, and the association of qualifiers (modifiers) with each of these parts. 

It is outside the scope of this paper and the conipetence of the writer to dis- 
course on this topic. It is cited as a widely examined example of the fact 
that humans develop and use organizing structures of variables, and relations 
among variables, in behavior that operates with wide varieties of content. 

In this context, another point can be made about the difference between an 
external format and an internal format. The external format of sentences is 
expressed' by written rules about grammatical "co)"rectness" . It should be 
obvious that individuals do not literally incorporate these rules in the form 
of these verbal expressions in the formulation of their sentences. The verbal 
expressions of rules may guide the selection and implicit editing of sen- 
tences expressing cognitive content, and thus assist in learning to criterion 
performance. But the implicit formats for the "grammatical" construction of a 
, sentence come to operate at behavioral levels where the structure of a verbal 
rule, such as "one should not split infinitives", is as irrelevant and mis- 
leading as to say that the mechanical design of a switch contains the "rule" 
whereby it will open or shut. If this kind of analogy is used (and some of us 
do use it), the context should be clear. 
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One learns to operate with behavioral models not by verbal descriptions of 
the models — except at very preliminary stages but by the exercise of the 
model in behavior. It is not clear that a large body of educational and train- 
ing practice recognizes the difference -- as well as the relationships — be- 
tween external task formats and implicit embodiments in behavior. 

For the time being, let us leave psycholinguistic models with the scholars. 
They have_ given us, however, an example of the notion of task format in a 
respectable field of inquiry. Their findings about the processes whereby 
implicit cognitive forjnats develop and operate may help us better understand 
these phenomena in other than linguistic behavior. 

"Understanding" a Task 

Except where the manager must serve the role of teacher, it is generally recog- 
nized that management" of workers does not require the ability to perform 
their various tasks at acceptable skill levels. The unsettled question then 
becomes "What should a manager know about 'the job-tasks he manages in order to 
assign, coordinate and evaluate?" A reasonable proposal is that he should be 
able to Identify and describe the formats of the tasks of the people who report 
to him -- where these descriptions take the structures that have been discussed 
in previous pages. This is the structure of the task variables in input problem 
situation, output response and reference entities and attributes. The details 
ot the processing rules and operations that translate input problem to output 
solution, although obviously essential to performin g the task (and to teeiching 
M) are secondary to a useful understanding of It tor management purposes. 

If a working knowledge is to be extended beyond format structure, however, the 
most useful information about the processing function consists of the strategic 
tradeoffs in the task criterion. In other words, what is the task operator try- 
ing to optimize (or should upLimize)? The dispatcher was trying to optimize 
the matching of a resource skill with a resource demand but with some minimum 
ot delay in time-to-repai r multiplied by the severity of the problem. Even if 
objectively measurable, these sets of parameters are not linear nor entirely 
independent of each other, so that "optimization" may, under heavy load, have 
to be substituted by a satisficing criterion. 

If the manager has an operational awareness of (a) the task format; (b) 
criteria of effective as contrasted with ineffective levels of performance and' 
• (cj the strategic tradeoff variables to be optimized or satisficed in perform- 
ing the task, he will often have more explicit knowledge of the task than the 
incumbent on the job. 

This kind of understanding can be functionally useful to professional persons 
whose work interfaces with other expertise. It should also be useful to ^-hose 
engaged in cross-training of personnel for career guidance and in the struc- 
turing of performance tests. Its significance to those engaged in the design 
of tasks and. task supports has been mentioned. 

This kind of information about tasks may be contrasted with job-task descrip- 
tions consisting of a list of work "functions" or of detailed procedural 
statements. 
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Conclusion 

Learning that is directed tcwarc:^ M.nrfc. raid nee criteria is a process of increas- 
ing order and organization from states of disov^der and uncertainty. The ex- 
pression of disorder is contained in the statement, "What am I suppoed to do 
and how do I do it?" The l6vel of a competence or skill is tested by the 
operator's coping with circumstances that he has not precisely 'encountered be- 
fore: the imposition of order on uncertainty and variability. : 

Depending on the nature of the task, the conditions of training and performance 
and perhaps on the cognitive capabilities of the operator learning and per- 
formance may occur at more than one apparent level. To the extent that there 
is variation in conditions between! one cycle of perfomiing the task and another, 
the operator learns not merely a one-for-one stimulus-response -relationship. 
He acquires a format of processing variables. A mathematical equation is an 
extremely crude and unsophisticated analogy; ^the 'value of variable x added to 
the value of variable y yields the value of variable z: that is, "Add the rate 
of the rower downstream to the rate of the current of the stream to determine 
the rate at which the boat is moving downstream." The process of learning -- 
at least under some conditions enables subjective "formulas" to be developed. 
The requirement is some pattern of regularity in the variables of the problem 
presented to the learner, but variability in the. values that the variable take. 
The clod does learn the physics of operating a wlieelbarrow over rough terrain 
in the sense that he incorporates it in his behavior with the wheelbarrow. 

A conceptual construct has been required to account for the everyday knowledge 
that people adapt the literal content of their experiences (i.e., what they 
have learned) to variations in problem situations not literally equivalent to 
previously experienced patterns. The notion of implicit task formats seems to 
help account for these adaptations. The notion aUo accounts for some of the 
phenomena of improvement in task skills that are recognized on a subjective 
basis but not readily fitted into traditional cognitive schema. 

In psychological terms, one should not think of a task format as a mechanism, 
or produced by a specialized "format generator" in the nervous system.. Rather 
it should be regarded as a phenomenon having, like any other product of learning, 
variety in kind and in effect. It .is to be hoped that the label "format" is 
not taken too literally or mechanistically. It is d. logical construct, like 
the constructs of attitude and motivation, which are tested not for truth as 
perceivable entities, but for utility. in generating hypotheses about practical 
as well as. theoretical matters. 




In .the practical . community where objects and procedures are designed for human 
use, and where training is designed for prbgrarmiing these procedures into human 
operators, the notion of task standardization may acquiYe a broader base. Some 
designers are guided by the philosophy of giving the operator a completely un- 
structured work environment with the intent of giving him freedom and flexi- 
bility. But with repeated practice, the operator will develop spontaneously a 
structure for coping with constants in variable situations in a constant way. 
The risks in freedom are slowness in reaching criterion levels of task effi- 
ciency and the probability that a spontaneously developed task format may 
usually be poorer than one that has been deliberately designed with the full 
range of task purpose, task environments and the properties of human behavior 
in mi,nd. On the other hand, a few gifted individuals mcty spontaneously develop 
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implicit formats of their own which, at least for them, are superior to their 
use of explicit formats designed by others. The risk in designing an explicit, 
external format for a task is that the structure may be inappropriate, or that 
It will unduly tyrannize and stereotype a task which should be performed with 
judgment and flexibility. But thes.e are the well-known tradeoffs that apply 
to any form of standardization whether in tasks or in automobile mufflers. 
Only to the timid or uninformed need a tradeoff be a standoff. Any finite 
solution should be aimed at a statistical expression of success, not an abso- 
lute one. 

In any event, the clear recognition that human operators can and do generate 
behavior that abstracts and copes not only with data, but data variab les in 
procedural and conceptual tasks, as well as in tracking tasks such as aiming 
and driving a vehicle, offers design and training concepts that can facilitate 
order in human task learning and performance with less than a straitjacket 
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GOAL PROJECTION 



A specification or characterization of a desired or 
satisfying state of affairs which a system seeks to 
acquire or maintain as shown by the alternatives i't 
selects (or avoi^ds) in its choice-making behavior, 
and towards which it is energized in its drive levels. 

The goal projection or image is the reference information with which a system 
compares a present or planned state of affairs. If on comparison a difference 
is found, behavior is modified accordingly in direction and/or mag-iitude. 

In an automatic system, goal image or purpose is implicit in the cybernetic 
control mechanisms. In human systems, or systems that include a human opera- 
tor, purpose and goal state concepts may be both implicit and explicit, and 
may be- partly irfdependent of control mechanismc. 

An inanimate system may seek an equilibrium level and direction of activity^ 
that comprises its operating program with changing state of the environment. 
An animate system may have a striving component for moving by means of task 
actions from a given status to an approximation of a reference goal state. 
The internal "reference" for the human operator is a goal image. Thus, a 
goal image serves as the embodiment of a criterion for terminating a task or 
segment of work or mission, and terminating it with an experience of'some 
degree of success or failure. 

Different goal images may apply to different levels of activity; e.g., task, 
duty segment, work cycle, mission. 

Examples 

t The recruit with a clear picture of what a clean rifle should look 
like, who continues cleaning it until, from all suitable points of 
regard, its appearernce meets this criterion. 

\. 

t The workman who fabricates a part 'to within his idea of what the 
inspection tolerances should be. 

t The mechanic who services an engine until it looks, sounds, and 
operates well according to all its performance variables as he 
knows them. 

t The pilot flying "straight and leveT" on instruments in a straight 
and level maneuver. - ^ 

Functions of Goal Image 

The goal image serves as both a steering and a power reference for moving from 
a present or projected state of affai^rs into a projected route of action. Goal 
information is operationally meaningful insofar as the behavior of the system 
is not completely and rigidly programmed or proceduralized; i.e., where the 
system has options for activity in kind and amount. 
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Goal Projection 

Where the system operates on a performance criterion rather than on a fixed- 
jifl^ the goal -data indicate a point of work completion 

alld ;«eset.^° ^° determine the subjective condition or"c?osure" 

Forms of Goal Image - ' " ■ 

* mV^^'.^nf^i"'^'.°^'"^f "^^^ ^ n^ental p'ictorial 
picture of an idealized or satisfactory terminal state. The image mav 

• than o'thVrs/'^"'"' '"^ '"'^ ^'''"^ °^ some variables - 

I'ome^w:;ter?:nge^ o^va^ies' °^ ''''''''' ''^''''^^ '^''^^ 

The goal picture may consist of indications of "progress" from a 
present position on .one or a group of criterion, variables, rather 
than terminal values. 

Behavior Tendencies ■ ^ 

Disregard Of goal image. The student and novice operator often' try to start 
■variab?L n'^^'"^ procedures and manipulations rather than goal^ and goal 

H?c!- ■ u '"'"^ instructor must insist on the student learning 

„?n\o^th?Sechanrcs%'Te^?a°:k^°^^ '''''' ''''^ the trainee-trSl?^ 

Deteriorating goal image . Goal images may become tentative and changeable 
Change is often in the direction of deterioration. This decay may occur in 

a hiSrr! f ''f^ ^'^^"^""9 "^'^ and expeTtise'^demanded for 

?n5 S I 9oal confront a ow energy budget and low student ability Sodal 
influences are well-known modifiers of goal levels or levels of asp ration 
The goal image may vary markedly within and between work cycles uSder ihe' 
ZlTT. °' physiological states, stress, work context a^d other factors 
These changes may be adaptive and functional in some cases. Goals that are 
spontaneously" degraded are not likely to be the best operating adaDtatinn. 
however, so that training may be necessary to guide the Spe?alo? ?n priori Eies. 

Disappearance of goal image. Routi nization tends to blunt the claritv ;,nH 

eTaTbXiSr 'H^i^'^'h^' J^'V disappear inEo' aStoml't^c'^r ' 
ceaura behavior. This may be adaptive in the sense of requiring reduced in 
formation processing activity if the. automatic perforLnc^i t^eLt acceSt^ 

luit ?Sn'drt?on^"^s3^s\:%5:f ^ ''^'^^ °- 

Training and Performance" Strategies 

The<;^strategies set forth below reflect not only the operational factors involved 
that' ?°thp'"n °^°Ph^""l and psychological biases of the author 0 e s 
that If the operator does not continuously strive to do a "better" job !- 
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according to his own lights, as well as those of others.-- his performance 
wiyll tend to degrade, and he will oscillate just above and below externally 
acceptable levels of performance and, in a clutch situation, he will fail. 
Another bias is the belief that a job or task can remain interesting after 
acceptable mastery has been achieved only if the level of aspiration for the 
perfarmance of the task always goes somewhat higher than the actual achieve- 
ment level. Even intrinsically dull tasks can remain bearable if the opera- 
torcontinues to try to beat his own record. Obviously, the operator's work 
environmsnt should not penalize the achievement of hi'gher levels of perfor- 
mance; at worst, the work environment should be indifferent to such achieve- 
ment. 



Clear goal variables . Training should emphasize (and test) the student's earlT 
acquisition of goal varieties. , Where possible, emphasis should be placed on 
concrete pictures of good goal states "that embody, the desirable values among 
. the entire pattern of goal variables associated with the task. The goal 
images that are presented should be realistic, not idealized ones that can be 
achieved only under ideal ized conditions that never occur. If goal images , 
presented are not realistic -- realistic in terms of a skilled operator in ^ 
representative conditions - the student will become disillusioned and 
cynical of the instruction. The student should be taught- to identify the ' 
variables- that comprise "a goal state. 

Tolerance ranges. The student should learn the range of values that would be 
operationally acceptable on each criterion variable. If this is done, the 
student should be able to distinguish values on each variable that are barely 
acceptable and those that are unacceptable; he should also be able to dis- 
tinguish limits in terms of the" composite goal picture as well. If there are 
operational tradeoffs between goal variables (e.g., speed versus accuracy) 
the acceptable range within which these tradeoffs can be made should be ■ ' 
' clarified. 

Reliably clear pictures of goal states, goal var-iables and their tolerance 
ranges should precede training an the performance of the work. Thisjis sound 
training strategy because it enables the student to -serve largely as 'his own 
feedback mechanism, a psychologically good 'arrangement in that it centers 
initiative in the student rather than leaving it entirely in the instructional 
mechanism. The strateg.y also 'induces' achievement motivation in the student 
who may- maintain it after he terminates formal instruction. (Contrast this 
with the s.tudent who continues improving only under the direct guidance of an 
instructor.) Application of this strategy makes learning purposeful -- -in 
terms of objectives -- rather than mechanical. 

..Priorities and contexts. Goal aspirations may have to be compromised by opera- 
tional exigencies. The operator should, therefore, know which criterion 
variables have the highest priority for sustained integrity and which may, 
when necessary, be degraded with least operational penal tyu for given kinds 
of missions. Such- knowledge enables one kind of rational judgment to. be 
exercised. '-.It may also enable a mission to be completed which othenvise 
would remain undone; if there seems to be no choice for th'e operator to go 
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ahead even in a degraded mode, he may flounder or cease to do anything at all. 
It is better, for/-example, to demolish a car than to kill or maim a pedestrian 
a choice which 'apparently some motorists have not been trained to make 
correctly. 

A concept of operational (and ethical) p-riorities among a collection of goal 
cnt€na is essential to the higher levels of work performance. In fact, 
strategic ruies" for selecting goal priorities make a large number of the 
strategies described in this handbook as procedural, seem almost trivial by 
contrast. In simple terms, the issue is what is more important, and what is 
less important to achieve in a task or mission. By being relatively specific 
on such matters, the operator himself becomes an active strategic element in 
the enterprise by being able to maintain a control initiative. In actual 
practice, how often does any worker get consistent instruction in qoal 
priorities? ^ 

Rational c ontrol over impulse . To the extent that the operator has a clear 
picture of goal states,- and is guided anil driven by them", he has a basis for 
rational control over emotional impulse and the tendency for expedience at 
the moment in other words, he has a basis for "localized optimization". 
When the operator .is under strong pressure from an internal or external source 
any device for the control of emotional impulse, and/or proper direction of •' 
impulse as a source of energy, will have operational payoff. Obviously, -such 
devices have the highest payoff frequency where the stresses are greatest and 
most frequent. (Note that evefi boredom is a' stress!) Indoctrination and re- 
furbishing of the subjective goal image, therefore, has a corlt.inuiTig stra- 
tegic value for operator performance. 

Comment 

These "strategic principles" for training and performance are indeed self- 
eviuent and are probably known in one form or another by the layman It is 
unfortunate, however, that such knowledge seems so rarely to be put'into 
systematic practice, even by specialists in the training and educational 
arts and by managers. 
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SCAN - DETECT , ' . 

Is soiiie thing tjher'e? • ' ^ 

Procedures and mechanisms for sensing the presence of 
a cue or condition requiring that some form of' action 
, be taken by the system. The action may consist of 
' getting more information about the cunditfon detected. 

Detection requires discrimination uf an 5'-:ti-in-sti.mu''at1ng cue from some 
background. What is detected may be occurring in the environment or within 
the sy-5tem itself. 

Detection is ordinarily a positive, consummtot*y response to the activ'ty of 
scanning . "Scan" and "detect" should, therefore, be linked activities and 
th^ are treated as such here. Since effect'iV,e detection is largely a 
furtttion of effective scanning, major parts of the discussion in this section 
deal with scanning conditions and strategies." Scanning is defined as the 
moving exposure of a. sensing device to a field of possible signals. The 
pattern of exposure may be systematic or rarjtiom, periodic or aperiodic. 

■0 ■ ■ 

The nerms scan, detect, search, identify, and interpret tend to overlap 
functionaTly, distinctions among them are somewhat artificial. .Subtle dif- 
ferences between search and scanning/detection, as well as between detection 
and identification, are sugyested by the following example. The operator may 
be looking for a needle in a haystack, but he can identify the needle when ha 
sees it, even though he engages in scanning and detects the glint of the needle 
before he identifies it as the object he seeks. In performing a search func- 
tion, ^the operator has a particular objective in mind. 'Detection is the 
sensing of an irregularity; in a stimulus .context oi^dinarily, detection calls 
for attention which leads to getting mere inFornation that enables identifica- 
tion. An automobile driver, for instance, iray detect an irregularity in thq 
"texture" of the roadway ahead and haq'.n an trvasive maneuver before he has 
identified the irregularity as a polhole, an old tire tread, or merely a 
shadow. , 

Examples 

Scanning and detection occur* in a large variety of .i:ontexts . 

J 0 Jhe indicator in an enclosed environment s^'Ch as d cockpit. 

0 An open field in a natural envi roninent , such an aircraft in the 
•open sky. 

® A bounded field in an artificial represente^ti on cf an environmen-t, 
such as a radar scope. 

0 An open field with a complex ground in which a figure is to be 
detected, such as an enemy snipt'- conceal ed in p^rt bv foliage 
and other terrain features. 
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Scan - "Detect 



A relatively closed field in which some property or attribute of 
one or more objects should be discriminated; e.g., a blackened 
wire tn an electronic assembly. 



• A collection of symbols; e.g., text which is being proofed for 
errors, a wiring xli agram which is scanned for an uncompleted 

^conn^ctioci. 

-In urban traffic, ^faipt whining sound that grows louder and is 
identified as a porl'ic6 'or' ambulance siren. 

• • A slight change of odor in a. house which is identified (once atten- 
.- tion is focused upon it) as gas from a leaking stove, or as a 
burning roast. ' •. ' " 

• An open boc|y of water in which buoys must be detected that indicate 
shoals, hazards, and the channel. 

Operational Variables' and Principles 

• ■ The lag or cycle time of the scan and detect function/device must 

be less than the cycle time required for the stimulus to be detected. 

•■ The greater the contrast between the^timulus to be detected and its 
background, the greater the reliabi lity of detection. 

• Some scan patterns and frequencies are better than others for 
detecting given kinds of signal patterns. 

• What the human operator detects is related to "set" or pre-established 
response tendencies. More simply, we tend to notice what we expect to 
see, what we are looking for, or what we are attending to. 

Discussion 

In scanning, both the motivational stimulus and the ongoing stimulation for 
what to do next are of internal origin to the operator, and are sustained by 
internal stimulation. During much scanning time, there is no response from 
the environment that serves as a cue for specific adjustmental action. Unless 
the operator has been aroused by an emergency state — such as pain, hunger, 
fear, an alert for some incipient threat — the human mechanism tends to 
avoid scanning. Consequently, we should expect powerful behavioral tendencies 
resulting in perfunctory scanning. During periods that the operator interprets 
as steady state" conditions between himself and the environment, the scan 
function will tend to be aborted. Strong motivations to reach a conceptualized 
or perceived goal reduce the quantitative and qualitative scope of scanning. 

These phenomena are normal selective conditions of human attention which are 
generally adaptive, but may at times be maladaptive. 
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Scanning, as a form cf infon.Btio.i p ocessing zaac msi be $>upportec! almost 
entirely by internal activities, is pj/**a^)3 mo -'e. vulneraUe to degradation 
and poor perfonv.ance than any othe^^ diaieission m 'cask r;;;c jirenents. In 
scanning, the beha/iora] f.i.incip!c. or f rc^ej^.^e minnnizatlon of information 
handled per activity haS tne few^.;t c '^''-.>t. .viik: factors in tha ti??k*environ- 
ment on which we can depend for rejirou;.t/i o\ the degraded function. 

Behavior Tendencies 

The following tendencies occur with <^xce:idei cxf^ericnc^ over number of task 
cycles, and also with experience? within a given Vx^^ cycle. In many respects, 
the tendencies are adaptiV^^ gehe>a]ly leading to reduced processing activity 
by the operator during task performance. In other respects, the behavior 
tendencies cited may be maladaptive, tending to reduce or delay the perception 
of potential crisis conditions. 

Sampling becomes s tanda)'dizod. With extended experience , . tlie operator tends to 
standardize: whatHRe pays** atterition to and when; the sequencirg of what is' 
scanned; and the classes of cue which are sampled during scanning activity. 
Often, the operatof^ standar^dizes what is no^ looked for ~- and the cue may 
escape detection, regardless of its inagM'tuJe. Cues \j'hich initiate scan 
maneuvers may also become standardized. 

Many scanning sequences, which are deliberately invoked early in task learning, 
become habitual and virtUcJlly involuntary wi t!^' i'^pe..'^ted experience. This 
tendency is adaptive i^t scanning habUs c'o.itinut to re^adequate to the work ' ' 
conditions, fcind if alerting signals are detecuvj as. such,. Since scan and 
detect must be a sampling activity, it is nut likcily to be succe^ssful in each 
and every environment, Failure probabilities in cvianged conditions and envi-^ 
ronments, and the cost of failure conjoquerices , may r''se to unacceptable opera- 
tional levels. In such a situai-ion^ "rede.^n":*' of t-o scan and detect functioji 
becomes essential Scanning app^op>^fate to lvVj -rt'^.v; of situation may be quite 
inadequate 'in other types of'^sUuat.ions . For fxai':^: * the kind of situational 
sampling that is appropriate to ingh speed L^jrr.pir:/.: driving is mostly inadequate- 
for driving in dense urban traffic, ^nd vice verFvi. - 

Sampling ^scop;^ and rang^ bocoiue s^uon JaiXii zccK i^ibst teiios to establish not 
only what is "sample J7T^urTfso^l ]e^^^ wnich f.ues are sampled. Some ^ 

drivers, seem to fix attention a standard d1st'*\n.,f: al>'':ad, and only upon their 
intended path; some sample only frcin the speodoiMC- ..Cff\ and generally ignore the 
otheK instruments (fuel gauge, tachomel.-^r^ etc.), h/':n though early training 
may induce more or less c,pprof;riate scanriincj b.hdvior, scanning tends to 
degenerate and unless scanning motivation arvJ . abics aje reinforced, the 
•function remains inadequate. Scanning h^.uits (ocitermin ing wh.at is sampled and 
the scope of sampling as well) become firrniy er^trenched -^^nd inadequacies only 
rarely are subject to adverse feedback: aporbpriate r^canning habits may quickly 
degenerate even after a period of determined improvtment by the operator. 

In certain op^erations, scanning haoi*cs<ma.y be ;;'e<;i-urc'd and corrected. A look- 
out on a ship must continually scax u f.;c.. .ri:.v.d ..'.. .tor and report all objects 
sighted. If the officer on thec..rid9e sights r.i objout or vessel that the 
lookout fails to report, correctvve actiun is caKin to correct/improve scanning 
habits. 
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Progressive standardi zation of figure-ground. sampling . An experienced jungle 
fighter who has learned to scan the brush for evidence of an enemy may, there- 
by, become vulnerable to enemies in towns and villages where a different pattern 
of figure-ground relations mUst be sampled in order to detect danger. The 
motorist who has driven mainly on well-paved superhighways, fails to scan for 
potholes ,in urban areas on ci ty. streets ; he would reliably detect an object 
the .size of a potholg if it were on the open highway, but not in the city. 
-This tendency differs from the preceding one, only in emphasizing that humans 
acquire sampling habits specific to given environments of figure-ground relation 
ships and variables. 

Reduced response- to partial cues. Detection may be the summation of a number of 
cue samples obtained during a scanning sequence. No "one of the cues in itself 
may be adequate to signal the need for overt action, or even to justify further 
inquiry. A bored sentry, for example, may hear the. sound of a crushed twig and 
Decome alert for a few moments, but return to a demi-doze.- Several minutes 
later, the swish of dry leaves and the cessation of bird chirpings may "arouse 
him again. None of these cues' by itself is likely "to be sufficient to justify 
arousal and investigation or alam. Together, they should indicate' the approach 
?u ^ u .''^3'^^^iJ3,l absence of perceived threat reduces the tendency to link 
threshold cues that, put together in short-term memory, would sum into an aware- 
ness of a , condition requiring full attention and action. 

Projection of expectations. Under conditions of strong expectations, spurious 
or random cues in the environment may trigger incorrect detections and identi- 
fications. The more complex the ground and the greater the difficulty in 
differentiating figure from ground, and the stronger the. anticipation, the 
-greater the liability of spurious detections. This tendency is a case of the 
interpretive process interacting with the detection process. Thus, under strong 
anticipation of an enerny attack, many sounds and sights will be detected as 
cues that would othenvise fail to be detected. Circumstances of this sort can 
also lead to over-response. When a military unit is engaged in an assault on 
an enemy position, a tense soldier may detect movement out of the corner of ' 
his eye and turn and fire in that direction. Only instantaneous positive 
Identification prevents him from shooting a buddy. Similar conditions occur 
in driving. When a driver anticipates an engine breakdown, he will detect as 
unusual, sounds and car behaviors which are quite normal for the car and to 
Wnich he would pay no attention when he is not apprehensive. 

Scanning and rate of chancie. This tendency is an important subset of stan- 
dardized sampling. Scanning patterns and sampling rates tend to become asso- 
ciated with some prevailing rate of change. Scanning rates appropriate for 
driving at forty miles per hour are inadequate at a speed of seventy or more 
miles per hour. The driver who has spent several hours on a winding two-lane 
road at a relatively slow speed is at a disadvantage when suddenly accelerating 
to limited access freeway speeds. He is, in fact, in danger on his approach 
to the.freeway, for this reason and others. The same difficulty appears in 
the shift in scanning activity required in the area of small local airfield 
compared to that necessary in approaching a large metropolitan area airport 
which services jet, traffic. ^ r <^ 
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Tunnel perception . In tunnel perception, the field of scan is markedly reduced, 
and tends to be limited to the intended goal path. Tunnel perception is an 
extension of the normal conditions of attention v/hich restrict what one attends 
to in terms of a goal, or to the execution of a nighly practiced chain of 
responses. It is an extreme form of preoccupation, A variety of conditions • 
tend to create tunnel perception: fatigue; anxiety; extremely high, motivation 
toward a goal; preoccupation with other than the task at hand; extreme routiniza- 
tion; and repetition of a series of standard stimulus-response sequences. 
Physiological depressants (including al cohol ) , no doubt, also affect tunnel 
perception. , 

Decreases in effective scanning and detection behc:;vior may be one of the most 
sensitive means of identifying the impairment of an operator or of his skills, 
(Impairment in short-temi men'ory capacity is another.) Because scan and 
detect functions are generally tested by relatively rare events, this impair- 
ment in scan-detect behavior may not be observable in the operations of the 
individual operator, but may appear only in statistical ensembles of task 
samples. 

Performance angi Training Strategies 

The way to train an operator for alertness and assure that alertness will 
persist indefinitely after formal training is not clear. In Infantry training, 
a soldier with his rifle loaded walks down a trail in a forest. He must 
instantaneously respond to targets that appear suddenly. He must identify 
these as friend or enemy and if enemy, fire immediately. 

In nature, animals deficient in scan and detect mechanisms (commensurate with 
the speed and power of their fight or flight capabilities) tend to have a brief 
life. In human societies, such selective procedures are less ruthless. More 
compensatory mechanisms tend to keep the occasional bobble that is due to in- 
attention from becoming a. catastrophe. 

It is clear that the learning of effective scanning behavior requires active 
and continuing cooperation of the student and cuntinuinn initiatives of the 
student/operator. He must not only acquire effective scanning and detection 
habits,* but also recognize various conditions that lead to deterioration of 
these habits so that he can compensate for them. 

The following are some training iinpli cations based on the liabilities cited 
above. - 

Cognitive mo de^^ of threat . In opei^atlng any vehicle, there is a zone of 
threat involved, bate driving consists, in part, of keeping as large a peri- 
meter of safety against potential threat as practicable. This perimeter is 
not symmetrical; it is greater to the front than on the other three sides. 
The higher the speed, the greater the boundary of the safety zone. If one 
must stop quickly, the rear presents a hazard. The skilled operator, perhaps 
by sampling every few seconds, is aware of the location of other cars including 
oncoming traffic. In case there is a need for a rapid emergency response, his 
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awareness .of the location of "competitors for his space" provides him with ' 
the greatest number of options for maneuver, and the information for choosing 
instantaneously. By continuous sampling from the entire region of his task 
operabions, he is aware not only of the potential threats, but of his actual 
response options. This might well be thought of as an operational model of 
\ tne lifespace of his task environment. The greater the driver's skill, of 

course, the better he can cope with constricted lifespace — down to some 
limit. 

Highly versatile, programmable simulators may be developed which can generate 
rea.listic situations that enable the operator to achieve effective samplinq 
rates for various configurations in the environment. The relative frequency 
of hazards posed by the simulator would have to be far greater than what the 
operator can expect in real life. But he would be learning to cope with 
non-representative situations - to anticipate through scanning and detection 
and perhaps to avoid threatening/hazardous situations. 

A cognitive model of threat is a representation of the dynamic factors in a 
task that need to be sampled at some frequency and range in order for 
probable sur^^ival . Sampling that enables prediction of an event(s) generates 
the largest. range of options for coping with the event if and when it does 
materialize. 

A model of opportunities, the converse of a model of threat, is a "map" of 
the factors to be sampled by the operator in order to increase chances for ' 
a successful outcome. The map must include representation of events in 
real task time to enable the student to acquire habits of scanning and 
detection, rather than merely ideas about them. 

Scan redundancy. If time permits, befora committing himself to a response 
in a dangerous/serious situation, the operator should scan his environment 
several times for obstacles, or for competitors of the path he is choosing 
and for a potential escape route if his maneuver fails. Scan redundancy 
permits the operator to evaluate the relative motion of his and other vehicles 
JlnJnnl 'J^^y"'^^"^" performance strategy is especially significant when moving' 
from one^fie d or rate of action to another. Critical information, missed 
in one scan (sample), may be caught or summated in a second or third sample. 
Look before you leap." should perhaps be amplified: "Look twice before you 
take a long leap over a deep chasm." ^ 

, Scan redundancy may be especially important if the operator is^ in an impaired 
state; e.g.. fatigued, anxious, strongly goal-driven, affected by severe 
stress, or emerging from preoccupation with a particular matter. 

Counteracting b lind-habit . Since routine habitual activities tend to elicit 
tunnel perception, the operator must generate and maintain initiatives for 
scanning; he must scan the environmental field for liabilities which may 
not have occurred in his past experience. Occasional refresher trainina 
which inc udes programmed simulated hazards may help. Such training s^ 
however, likely to be both brief and somewhat artificial, hence of limited 
effectiveness. There seems to be no external operation that would slrle 
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as an adequate alternative to (either replacing or supplementing) operator 
initiatives. The operator himself must periodically check his owh scanning 
skill, and deliberately modify and improve it in order to prevent deteriora- 
tion.^ 

An/operator may go through the motions of habitual scanning and see nothing; 
if so, the detection process is not at work. The operator needs occasionally 
to test himself on this score. Noting that he has just looked at the rear 
view mirror, for instance, he may ask himself "What is behind me?" If a 
moment after scanning he cannot recall what he saw in the mirror, he is not 
processing effectively. A friend of mine was overheard telling an associate 
that he had driven to work that morning, as he had for years, but couldn't 
remember a thing about the entire trip. This was disconcerting to him; he 
decided to -revise his processing activities including scan and detect be- 
havior while driving. 

Awareness of impairment . Aware that he is drowsy, tired, or tense, the opera- 
tor may compensate by deliberately putting more effort into scanning to avoid 
tunnel perception. If he is unable to do so, he should take the operational 
precautions available to him. The motivation for doing so may be stimulated 
by information regarding the effects of failure to respond to the signs of 
impairment.' This is ona basis f6r highway safety ads on TV and in the news- 
papers that graphically and/or verbally depict consequences of drunk driving, 
not wearing seat belts, etc. 

Ambiguity and stress . Under conditions that combine ambiguity of environmental 
cues and operator stress, the operator may become dogmatic about what he has 
detected and apparently identified. The rational strategy under conditions 
of stress and ambiguity is to consider a conclusion as merely an hypothesis 
and "collect more data" by further scanning for cues that support or 
nullify the earlier conclusion. This strategy demands a high level of 
rationality, learning from a series of recent unpleasant experiences, or a 
combination of both.' The application of this principle demands an unusual 
form of professionalism on the operator's part. 

Peripheral scanning . In some time-constrained tasks, it may be desirable to 
scan a wide perimeter even though little time is available for doing so. 
Instruction and exercises can be .directed towards increasing the effective- 
ness of scanning with peripheral vision. Football and basketball coaches 
teach this. with apparent success. 

It is possible to detect an object peripherally without being able to identify 
it. It is also possible to detect an object in central vision but be unable' 
to identify it (due, for example, to lack of time, urgency, etc.). This 
phenomenon has been seen in tachistoscopic experiments. Under time^'pressure, • 
a driver who is unaccustomed to the use of a side mirror may glance rapidly 
over his left shoulder and return his gaze to the roadway ahead in a fraction 
of a second; this- activity may be sufficient to detect a shape but insuffi-.. 
cient to identify the shape. - Identification may not be necessary and may. 
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in .fact,, be counterproductive if split-second reaction is necessary. It 
is unlikely that any novice to a task will refrain from the temptation of 
spending the time to identify something which he has detected. The extra 
few tenths of a second he takes may mean the loss of an option for 
successful evasive action. The advantage of a few tenths of a second 
multiplied by many exposures can have a significant effect on accident and 
other failure statistics. 

In time-critical circumstances (and perhaps others as well), the operator 
should gather no more information about the situation than is necessary' 
for selecting an effective operationaV response or discarding an ineffective 
one. 

Gapsjn s canning . Like other behavior, scanning is likely to have discon- 
tinuities in effectiveness, gaps between fixations — especially in visual 
fields that are noisy and without reliable references. A set of super- 
imposed grid lines is an example of a reference for scanning. The scanning 
problem may be that of trying to detect crash survivors in a turbulent body 
of water from an aircraft. Two scanning strategies may be used to ensure 
scan "interleaving". One is to del iberately overlap segments of the area 
scanned. This principle is used in radar scanning for early detection and 
warning systems. A second strategy is to successively scan the same field 
with different scan patterns, one vertical sweep and once diagonally across 
the field. If the field has a definite directional texture, three scan 
patterns over the same field are more likely to be effective — one with 
the grain, one against it, and one across the grain. 
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What is it and what i=; its name?. 

Methods for characterizing a message by type or 
source, used ordinarily to recognize an object 
. or entity and apply a label to it. 

The definition extends to a variety of contexts in which the identification 
function operates. Identification requires a referencing action which pro- 
duces the name or same equivalent symbolic response to be attached to the 
sensed input. In human behavior, the content of this symbolic response 
may not always be explicit. One may "recognize" an individual even though 
his name is not recalled, nor the explicit situation in which you met. 

In automatic information processing, two sets of reference codes may be 
necessary for adequate identification. One reference structure may apply 
to the universe outside the processing system (for example, the name and 
address of the sender of the message). Tha otlier reference structure is 
to the physical and/or functional location of the message within the system. 
These two identity codes may require a set of cross-referencing rules' or 
codes. 

Discussion of the Behavioral Process 

The learning of an "identification" consists essentially of two parts: 
learning to recognize a pattern of cues that constitute a real or conceptual ' 
entity (which may be a process or condition as well as a thing, or the class 
to which the thing belongs); and learning the name to be attached to the 
entity and to associate the name with the entity, the entity with the name. 

The name can serve in short-term memory as a transcript code for the thing 
or condition conceptualized. This transcript or code may take up less 
"space" in short-term memory than might be required to hold the image of 
the entity in mind, especially where a list of items must be kept in mind 
for procedural processing. 

The learning of an identification combines che processes of acquired dis- 
crimination and acquired generalization. Within tfie context of a succession 
of experiences with the entity, cues are sampled wliich become at least 
sufficient for discriminating the entity from competing entities — again, 
withiii the experience pattern of the learner. A name or other symbolic 
gesture becomes associated with the cue patte.'-n. With extensive repetitions, 
only structural elements in the cue pattern sufficient for the identification 
(in the experienced contexts) tend to be used. This abstract of cues gives 
rise to the object constancy phenomena. It leads to progressively fewer 
cues being sampled for making the identification, down to some limit 
adequate for discrimination of the entity. The abstracting process fosters 
stereotyped perceptual response. The operator may have difficulty in 
actually "seeing the object as presented to him" in terms of actual sensory 
input; he sees substantially what he expects to see. 
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If an individual is required to identify individual members in a class of - 
members with which he is unfamiliar, another process seems to -take-p-l-ae e > — 
Consider an American visiting a Chinese city, ' At first, most of the faces 
seem alike; he has difficulty in remembering the distinction between . the 
doorman and desk clerk. In a few days or weeks-> this difficulty tends to 
disappear. It is as if he developed a new perceptual matrix for differ- 
entiating members^ within the new object class, and thus for identifying 
individual entiti'es. It is unlikely that learning a new perceptual "set" 
or perceptual, matrix can be facilitated by deliberate cognitive analysis 
of the new matrix. (Recall the aircraft Identification procedures 
developed in World War II which were effective when samples of the entire 
aircraft species were displayed for recognition of the pattern, but 
relatively ineffective when taughrt by analytic methods.) 

Every work environment will have several classes of entities, the members 
of which must be identified by the operator. If, as a student, the operator 
does not learn the identifications cf the tool's, objects, or operations -of 
his work environment, he is handicapped in learning work procedures and in 
remembering/following instructions that deal with these entities. That is, 
he lacks subjectively clear pairings of names and image with which to 
translate the words in the instructions into a picture of what is to be 
done, or to transalte the appearance of a work object into a name which 
can be linked to a verbalized Instruction. 

Behavioral tendencies * 

The following are, for the most part, normal behavixir tendencies which can 
sometimes be liabilities occurring at different levels of training and 
experience. 

Recognition despite noise . . The operator fails to, recognize an object pre- 
sented in the presence of perceptual noise, unless 'the correct conceptual 
matrix is presented. The perceptual matrix is the equivalent of an 
appropriate class set. Consider the case of a noisy radio channel over 
which words in English speech become barely recognizable. Some French words 
are spoken. Although the listener may comprehend French 'nearly as well as 
English, he tends not to recognize the French. Once some pattern of sounds 
becomes audible he may begin to recognize the French words. 

Differentiation of cues within an entity . When the identification of an 
object has been highly over! earned in experience, the operator tends to 
fail to differentiate cues within the identified entity and/or between 
members in the entity class. Failure continues unless and until a new per- 
ceptual matrix for such differentiations is learned, or invoked, requiring 
that such differentiations be made. An indifference zone in perception 
tends to developvwithin which variations in an object are not noticed. For 
instance, unless maintaining a clean car is a matter of concern, one 
normally does not notice the day-by-day variation in dirt that accumulates. 
Habitual recognition, leads to a reduction in sampling of presented cues. 
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Projection of expectations . Identifications of members of a class of 
entities can perceptual ly degenerate into the sampling of minimum cues 
that confirm a perceptual stereotype. Failure to differentiate members 
of the class results, of codrse, in failure to differentiate experiences 
from different members of the class. Hence, the operator is unable to 
profit from such experiences.. In order to reverse this state of affairs, 
the operator must be jnduced to -pay attention to cues he has learned to 
ignore. The stereotyped perception must be inhibited. Changed, stimulus 
sampling habits must be substituted. Even assuming cooperative effort on 
the operator's part, these changes can best be brought about in a new 
work environment which will not provide associative support for the old 
perceptual' habits. ' 

Attributes sampled . After considerable experience on a task, identifica- 
tions by the operator become limited to specific attributes of the entity 
essential to the task. If a clerk's job is to sort orders on the basis of 
the color of paper on which they are printed or written, only that feature 
of the orders will be identified; he will tend to ignore other attributes 
(e.g., kind of paper, written content, texture, and so on). Here too, if 
the operator's task is changed so that he must consistently note and 
identify an additional characteristic or two, task retraining virtually 
from scratch may be required in order to assure reliable performance 
and there may be occasional regressions after the new discrimination has = 
become psychologically "automated". 

Identifiers emphasized . The art. of the caricaturist consists in perceiving 
and overemphasizing the pattern of distinguishing characteristics of people 
and other subjects. The ease with which the person caricaturized is 
recognized suggests a general tendency for selective emphasis of shape 
elements. This selective emphasis on distinguishing characteristics has a 
functional value in retrieval of image content from memory, and suggests 
an hypothesis for training. Training for identification of members of a 
class of objects should include artificial enhancement of distinguishing 
cues early in training, and progressive elimination of enhancing cues as 
training proceeds. Caricature, in this sense, would be useful only where 
a large number of class members'must be differentiated and identified, 
and where it is desirable to have the student learn these identifications 
without the benefit of working contexts. The policeman who is required to 
memorize the appearance of a large number of fugitives is a case in which 
caricaturization would be useful. 

Identification without naming . It is possible to identify a perceived 
entity without recalling its name. ("I recognize your face but I can't 
remember your name.") If the name is also given, the operator may recognize 
it as belonging to the percept. Work situations exist in which it is 
important that name and percept be reliably and rapidly associated and 
interchanged. In such work situations, practice of recognition that re- 
quires, overt naming when the object is present, or the percept recalled, 
should be required in benign work circumstances. There may be large 
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individual differences in ability to recall these associated elements; 
some persons may be more prone to forget the namd, others to forget the 
image — , face — that goes with the name.. One can readily imagine the 
difficulties which would occur in an operating theater if a surgeon could 
not readily recall the. names of the tools he needs to perform vthe ■ 
necessary activities. 

Identifica^ tion by progressive subsetting . Scanning may lead to the 
detection and identification of a hand tool lying in, the grass beside a 
workshed. Approach and further scanning leads to identification of the 
hand tool as a screwdriver. Still further scanning permits the tool to 
be identified as a phillips head screwdriver. It is assumed here that as 
the operator approaches the object, progressively more cues become available 

In many cases, an object or situation may be scanned and subsetted for the 
primary purpo.se of determining whether the object justifies concern (that 
is, indicating whether the object is instrumental to some current functional 
or potential purpose) or justifies indifference and no .further attention. 

The example cited is not intended to imply that the logical order from 
^more general to more specific classification is necessarily characteri'stic 
Nof the psychological process. Rather, the sequence tends to be determined 
By^ the kinds of cues that are perceived and integra1:ed into the pattern of 
cues making up the entity as an object or condition for further interest. 
Gest^lt theorists have emphasized the integrative aspects of perception as 
fundaniental and essential. For practical operational purposes, adoption of 
an- eitKer-or choice, does not seem^ necessary. 

Studies of pattern recognition generally point to the conclusion that if 
entities (enemy versus friendly aircraft, for instance) must be rapidly., 
identified from minimum cues in any of various orientations, learning the 
entire pattern enables identifications to be made from fewer cues and with 
more speed than by analytic cues. But these results can also stem from a 
well-known principle -- the more closely conditions of the training stimulus 
simulate those of the task stimulus, the greater the transfer af training. 
(This may .not, of course., be the whole story.) In addition to learning to 
perceive the entire pattern aTl at once, analytic training has value in 
teaching the operator to test perceptual hypotheses — when time and oppor- 
tunity permit. general pattern or shape may be "identified" as an enemy 
aircraft in the s%r, the identification may be an hypothesis that requires 
some confirmation. Mhen fQse,Jage cues become visible, analytic knowledge 
that enemy ai rcraft- have a characteristic bend in the fuselage, whereas 
similar but friendly aircraft have a straight fuselage, can enable positive 
confirmation of the i.dentification. In World War II, American infantrymen 
in Europe never wore overcoats. German infantrymen did. The silhouette 
pattern at night provided the basis for enemy identification. 

This discussion suggests that identification of entities that are fully and 
unambiguously presented may profit from a somewhat different training 
regimen than identification of objects of features under conditions of what 
might be called "stimulus deprivation", i.e., where the object is obscured, 
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'incompletely displayed, or displayed at near-threshol d sensory values. In 
the latter condition, recognition at any stage should tend to be accepted " 
as an hypothesis, and scanning should be directed toward cues that confirm 
or deny the hypothesis. Di rected scanning would be based on the operator's 
ability .to apply analytic operations to the scan activity and to assemble 
partial cues into a test of a recognition hypothesis. Our discussion in a 
preceding paragraph about the use of the emphasized identifier would apply 
here. 

Task Strategics 

The following strategies assume that the task demands reliable and correct 
identifications, that recognition habits tend toward low levels of cue 
sampling, and that tests for a tentative identification are desirable. 

Cue sampling . Taking into account the risks and penalties in delayed identi- 
fications, attempt to get more than a minimal variety of cues from the entity 
in question before making an identification. Cues may include the environ- 
mental or behavioral context of the entity. Adding more cues increases the 
probability of correct identification.-- assuming the operator has an 
adequate reference concept or percept of the entity. In other words, ob- 
serve the entity with full rather than perfunctory attention. The cost in- 
volved may consist of greater effort or concentration rather than extended 
response time. • 

Stress . Cue sampling for identification's- may become especial/ly poor under 
•any of a variety of stress conditions. If an operator is aw'are of this 
tendency, he, may deliberately compensate for the liability by more intensi\?e. 
checking for confirmatory identification cues. If a patrol in enemy 
territory is under acute stress, deliberate efforf~tieeds to be made to 
differentiate the color of the uniform of an approaching soldier- so as not 
to shoot a friend. This strategy has a. tradeoff — not shooting a friend 
versus shooting before the enemy gets close enough to do damage. 

Forgetting the name or the percept . As noted previously, one may -forge4f^ 
the name that goes with th^. percept or forget the percept that goes with the 
name. The recall strategy ts to search memory for associative context. 
(The reader should refer to Recall Enabling Information (Long Term Memory) 
strategies.) 

An obstacle or block in recognition 'of an object, or in recall of a name, 
.may be learned and persist in future encounters with the object, even . 
though its name is learned again following each experience. A formal re- 
conditioning process may be necessary to assure reliable recall when such 
a block occurs. (No Freudian interpretation is necessary.) 

Tests of an identification . Although it is logically impossible to ctertify 
absolutely that two entities are identical, it is, logically possible to 
prove non-identity of individuals. (The reader may recognize the same 
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logic in the null hypothesis in statistical reasoning one cannot prove 
the nun hypothesis, because one cannot logically prove from finite samples 
"no difference" 'between populations.) Tests of blood type cannot prove . 
tha'. the samples of blood were taken from the same man; they can, however, 
prove that the samples were taken from different men. Thus, in a difficult 
ulentification, a single negative cue ma^ be given more weight in rejecting 
trie hypothesis of. an. identification than several positive cues may have for 
accepting the hypothesis of identity. There are, of course, limitations in 
the practical operational value of the strategy of negative signs. Unlike 
blood type, many identifying cues can be altered. The strategy should be 
applied with an awareness of facts and a sense of probabilities. 

Key differentiators . Some variables are more important than others in ° 
estaDhshlng Wi identification or in. rejecting a tentative identification. 
Key _variables-5in the classes of entities in the work situation should be 
learned and applied. Knowledge of secondary variables may support the 
testing of a tentative identification when correct identification is "important. 

If th® analytic process, must^e used in reaching or testing an identification, 
categorization strategies may be;^ applicable. . ' " 

Training Implications ■ 

Percept and name . The learning and practice of some jobs is heavily dependent 
on reliable memorization of a large number of identifications. Medicine is an 
example. The student must' learn to recognize the appearance of-the entity or 
condition and to distinguish it from the .appearance of other similar entities. 

■ He must also learn the appropriate name or symbol for the entity and differ- 
entiate it from othetr names in the vocabulary. He must learn- to think of the 
object when he sees or hears the name, and think of the name when he 'sees the 
object. The operator may have to recognize the object in a variety of.per- 
ceptual and operational .contexts, sometimes in difficult-perceptual environ- 

. ments. These constitute variables in effective learning for identifications,.* " 

To these variables, we must add individual differences in the ability to • 
learn and remember percepts and in the ability to learn and remember names 
and other symbolic labels. If the' student has a deficiency in either of tfiese = 
abilities, his training should clearly have a different emphasis -'- assuming' \ 

■ that training is to be fine-tuned to. a well-defined set of "identifications", 
as a working task vocabulary preliminary to or concomitant with learning of 
task procedures. If the conditions 'of training impose a heavy requi rement 
for verbal learning as practice for task performance, this reqjjirement is 
imperative. > (It is also a requirement for useful conceptual learning --* in ' 
this type of material, "identification" is the equivalent to "definition of 
the concept".) . 

Some individuals with good verbal memories may be able Lo'learn the verbal 
content of procedures relatively quickly and may, therefore, do well on early 
classroom tests. Deficiency in the translation of verbal content into per- 
ceptual concepts permits individuals with poorer verbal memories, but better 
perceptual and perceptual-motor memories, to overtake those with good verbal 
memories in actual task performance. , ' ' 
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If computer assisted instruction has a graphic capability, it becomes an 
exeelleot , instructional device for adapting individual differences in rate 
and kind of learning of the identiftc^tions associated: with a task, and 
for achieving adequate" criteria of le'arning, at least outside the opera- ^ 
tional context of the actual wprk itself. . . 

Enhanced distinguishabil ity. . The distinguishabil i ty of the percept of the 
object caiji be enhanced by simplification and emphasis on the object's 
primary cnaracteristi cs ; i.e., caricature. Diagrams in many modern tex.t- ^ 
books follow this principle of selective enhancement and simplification. 
y(Il lustrations of this sort require a special skill, guided both by technical 
knowledge and communication^ ins ight.,),„ , . 

Caricatures ^.shoul d probably be used after the student? has been initially 
exposed to samples of the Veal " object; they should be discontinued as 
tfie student ..becomes able to reliably and Automatically mat^e the identifica- 
tion (aloag with identifications of the other objects in the seit to be 
learned.) It is possible that the use of an artificial caricature in train- 
ing may establish a learning and recall strategy for the student which he 
^wiTl carry. .into operational activities. Thus, it could serve as a supplement 
to learning the identifications of entities in the real work-situation. 
Learning i denti fi cations in the actual work envi ronment is ideal only if 
the right conditions prevail indi vi dua.lized tutoring on object-name and 
name-object with immediate knowledge of results and correction as necessary, 
continued for the many trials that lead to dependable recall. 



Contextual cues for identification . The method used to learn identifications 
'can make tjheir recall highly dependent on contextual cues. Training in 
context is efficient and effective when those contexts are present in the 
work situation. Keep in mind, however, that ^ high degree of practice tends 
to lead to minimum sampling of presented cues before an "identification" is 
made. If after such trai-nlng and experience, .'the operator must make the 
-identification in a strange con-text, he may be unable to do so reliably. 

Thus, if identifications must^be made in a large and somewhat unpredictable 
range of contexts, practice oh absolute (as contrasted with relative or con- 
textual) identification is prescribed. Identification should be learned in 
a minimum context (excluding se'condary matters, except for other members of 
the set to be discriminated fr'om each other), and in- a wide range of 
randomly sequenced conditions. The geol'ogist, archeologist , 'and detect-^ve 
share this kind of learning and performance problem. 

Interpretation and identification . The operator builds an identification 
from a number of contextual and inferec^tial cues. An inexperienced diagnos- 
tician makes a number of tests and, by a deliberate inferential process, 
.reaches a diagnosis. An experienced practitioner recognizes^ the pattern of 
symptoms all at once and identifies the trouble. In computer jargdn, this 
is equivalent to translating a computational operation intd-.a "pattern 



as a whole and matched with a pattern in memory to which a name is attached. 



matching" operation. The pattern 




more or less 
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Training has the option of • mi mi eking the extensive array of work experiences 
so that a large proportion of the neophyte's tasks are performed by identi- 
fication mther than interpretation. Costs of this kind of training in- 
crease as The number of wofk si'tuations is. increased. Expenditures may be 
justiffed if the mission is highly significant, and if task performance 
time is operationally critical. The lunar space-missions are examples; 
training was 'undoubtedly directed to the recognition and identification of 
troubles, insofar as condi'tions could be anticipated, rather than to time- 
consuming deductive and interpretive processes. 

There are operational tradeoffs between the identification mode and the 
interpretive mode. Spontaneous identification may be based on inappro- 
priate cue sampling and on irrelevant contexts. The interpretive process, 
although slower, is essentially one of hypothesis-forming and hypothesis- 
testing and tends to be more careful and sometimes more accurate. If time 
permits, and mission importance justifies, the best arrangement is redundancy 
of both, using spontaneous identification in creating the hypothesis and 
applying the interpretive process to test that hypothesis. 
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What does it mean? 

Rules for translating the symbolic context of 
a message into a reference or meaning, usually 
by 'the addition of reference context from within 
^ the message itself, but at times from outside 

the message. 

Interpreting requires response to a pattern of cues, and applies to events 
or conditions that go beyond input data or symbols as such. Input data 
are only a part of the total information required fo^^ interpretation. 
This is the way we differentiate interpretation from decoding. 

An interpretation is an inference about a condition, state of affairs, 
or source of dat^. 

Examples 

» Automatic analysis and "recognition" of an English word as a pattern 
contained in the physical wave form of an utterance. 

9 Human conclusion that the unannounced approach of foreign aircraft, 
detected and identified on radar screens, means invasion and war, 

• Human conclusion that a given pattern of symptoms signifies that a 
system failure must be caused by a programming error rather than a 
machine failure. 

• Language translation from Greek to English expressions. 
Process Varidbles . * . 

• Degree of statistical certainty regardi-ng correctness required of 
the interpretation. 

• Amount of redundancy in the form of context available in. the message, 

§ Range of variability among elements in the pattern to be determined. 

» Proportion of irrelevant transients in the message which act as 
noise to interpretation. 

A Number of elements necessary and sufficient to match with a reference 
set ("dictionary") of meanings or interpretations. 

• ' Number of alternative meanings or identification:, in the reference 

set available fur the matching effort. 

9 Opportunity for interpreter to luery message source for additional 
information for testing hypotheses about an interpretation. 
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Discussion 



Inducing or projectinq a "meaning" on a stimulus puttern or on message 
components in terms of some reference context, is one of a variety of 
thumbnail definitions of . interpretation. The essential ingredients are 
generally those of responding to more than what is immediately present 
or perceived, and the capacity of the presented cues to signify sub- 
jectively a cognition of state or purpose. Some process of logical or 
material jinference is implied, and a translation of a set of signs con- 
ceptualized. The translation may be from a symbolic representation, such 
as text or diagram, to physical representation or action. Or, the trans- 
lation -may be in the reverse direction; from a perceived actuality to a' 
symbolic representation in thought. A meaning^(in the sense of inter- 
pretation) may be expressed in what the operator thinks, what he does, or 
by a combination of both. 

Definitions of this kind are invariably incomplete, vague, or trivial be- 
cause a condition of awareness is connoted and objective definitions about 
states of awareness are likely to be vague or trivial. 

Examples of interpretation abound. The concept of the interpretive process 
may be among those which are useful mainly when they are rooted in some 
relatively definable operational context. Thus, language interpretation, 
map interpretation, radar scope interpretation, animal track interpretation ■ 
each suggests a fairly specific combination of environmental input, purpose 
and process which is pragmatically manageable. Note, however, that inter- 
pretation does not include (psychologically) literal- code translation such 
as translating English text into Morse code according to a fixed (to the 
operator) set of rules. When an operator, through experience, does find 
and appiy a set of rules for responding in some class of situations, the 
activity bpcomes either a matter of decoding or an identification proceed 
rather than an interpretive process. The rules may be imolicit and the " 
operator may not be able to articulate them. (This point' will recur later.) 

An interpretation is at least a subjective technique for the reduction of 
uncertainty presented by a complex of environmental signs that do not 
(according to the operator's programs) directly evoke goal-relevant 
responses, or that tend to evoke inconsistent or conflicting responses If 
we may anthropomorphi se, it is as if the human abhors ambiguity, at least 
in its subjective state where ambiguity is translated into uncertainty. 
Interpretation is an anxiety reducer. (A second class of anxiety may 
follow one s knowledge or belief in knowledge of a set of conditions— 
that is. what to do about it - but that problem takes us into decision- 
making.) Anmsight is an interpretation, and tne phenomena and strategies 
?ab°e^here ^' psychological literature on insight are directly appli- 

The analyses of decision-making, interpretation, and identification come 
together and overlap somewhat. (See earlier subsection. Identify ) With 
repeated experiences, adaptive processes in the operator tend to change what 
at one time is an interpretive process into an identification process This 
change does not invalidate the distinction between the two processes , 'nor 
the value of separate but overlapping strategic variables for each of them 
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The overlap does make it difficult for judges who must try to distinguisli 
one process implication from another in examining a task or collection of 
related tasks. The semantic distinction in itself should not be con- 
sidered important except to suggest operations and strategies of relevance 
to task performance, task design, selection, training, and so on. 

Behavioral Tendencies 

As in other behavioral processes, repetitions in a task context that require 
a class of interpretations {such as language translation or determining the 
intent of an opponent in a given kind of war game), lead to various regu- 
larities of sequence, s :ardardi zations of processing format, selection, and 
prioritization of cues which may potentially be attended to in a given 
situation requiring an interpretation. These may include cues available 
for "testing" an interpretation subsequent to making the interpretation. 

By definition, all of the cues necessary and sufficient for arriving at a 
conception of meaning in terms of a task situation, are not immediately 
present or, if present, are not given immediately to apprehension. There 
is opportunity and need for the operator to project informat^ion into what 
is given objectively. An interpretation is , therefore, an hypothesis, whiqh 
may be held with varying degrees of tentati veness or certainty. The hypoth- 
esis may be more or may be less valid according to some criterion independent 
of the operator's concept. 

The translation of a foreign language affords a convenient example of pro- 
gressive change in processing with practice. The novice begins by trans- 
lating words, more or less literally, one word at a time. As he progresses 
in vocabulary and syntax, he translates a phrase as a unit, buffering in 
short-term memory the several words of the phrase that he reads or hears 
until the phrase unit is more or less completed, with still more practice, 
he tends to use the clause or similar "unit of thought" as the context for 
interpreting a meaning of the text. Thus, he becomes ab-le to ""translate" 
words that are not enz:irely in working vocabulary because of their context. 
If he is translatvig from German into English, he must, acquire a somewhat 
different structural pattern (in short-term memory) for retaining elements 
in the symbolic pattern which yield a meaning. That is, he learns different 
message formats from tivose he is accustomed to in English. Ultimately, he 
may be able to "read for sense" without recasting the syntactic structure 
of the language he is reading into that of his native language. 

Quite literally, the student translator is building up an hypothesis of 
the meaning of a sentence and paragraph in the course of reading its elements. 
The test of the hypothesis is, "Does the sentence make sense with respect to 
some kind of reference to the context and the reader's experiences with 
sensible discourse?" This test may, unfortunately, be omitted in many 
interpretive acts of any kind when the operator is preoccupied with literal 
transcription of the component symbols. Perhaps this is why one of the 
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tests of sophistication in a foreign language is the ability to understand 
a joke told in that language, which would elicit amusement if told in- the 
native tongue. Jokes usually require the perception of an Incongruity 
within a pattern of references. 

Let us now focus on an outline of the factors in the learning process 
associated with interpretation in a class of environments. 

• Cue minimization . With continued practice, an hypothesis (conclu- 
sion or interpretation) is developed from fewer and fewer cues. 

• Standard test sequencing . Depending on frequency of failure data 
(hypothesis is incorrect"), standard test sequences in the examina- 
tion of cues are set up in order to test a tentative hypothesis, or 
to develop it in stages. Tests that fail to make differentiations 
tend to be eliminated. 

9 Short-term memory content . The number and kind of cues that are 
clumped in short-term memory before reaching an hypothesis become 
standardized. For example, a careful translation of a statement 
made in Chinese into English may not be initiated until major 
segments of a sentence have been completed, or perhaps until the 
entire sentence has been' heard or read. The sentence becomes the 
unit or pattern of cues from which the "interpretation'' of meaning 
is based, and this concept of the meaning is what is coded by the 
translator into English. 

0 Prioritizing of cue types . Some cues become more important than 
others in establishing an interpretation or a concept of meaning. 
The priorities are likely to be highly individual because they are 
linked to code hierarchies in long-term memory that are specific 
to the individual 's experience, and, possibly, also to his internal 
preference for one kind of code over another (e.g., auditory pre- 
ferred over visual, spatial, or kinesthetic). 

0 Inf 1 uence of reGency . Recent events in the operator's experience 
tend to dominate or at least influence interpretations, especially 
where the cues are ambiguous or incomplete. In other words, the 
interpretation is governed by the phenomena subsumed under 
behavioral "set". 

0 Dropping out of co gnitive mediat ing oj)era tions_. With high degrees 
of successful practice with the "in terpretTve content in a given 
task, the interpretive mode in terms of the active processes of 
implication — disappears and merges into an "identification" type 
of response. In other words, implicit verbalization and image-making 
become reduced as the operator's repertoire of standard response to 
standard situation is expanded (even though the situation is complex) 

0 St ereotyped repertojre . Differentiations in response, according to 
subtle differences in the input messages, may become reduced to a 
relatively small stereotyped repertoire. This is most likely to 
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occur if the operator must interpret under time pressure and - 
according to a quantitative criterion rather than a qualitative 
-criterion. Stereotypes may crystallize around the verbal code 
repertoires that are in the task vernacular. The reduction of 
alternatives from which selections are made, and the standardiza- 
tion of the cues for selecting an alternative, clearly are 
reductions in data processing complexity. 

Task Strategies 

Task strategies aimed at improving the interpretation process are, for the 
most part, aimed at avoiding or minimizing the behavioral liabilities already 
described. More positive and specific task strategies are likely to depend 
on the specifics of the task(s) and of the envi ronment(s) to be interpreted. 
In general terms, these will be the cliches of rational as contrasted with 
impulsive behavior form hypotheses tentatively; accumulate evidence 
through extensive sampling; correct hypotheses accordingly; test the conclu- 
sion by reference to some larger pattern or context in which the samples 
of data appear. 

The distinction between a training technique and a performance strategy be- 
comes highly blurred in these complex orders of behavior. 

Avoid stereotypical response . Reducing error probability in interpretation 
follows from reducing the tendency to stereotypical response. This avoidance 
consists of taking more perceptual samples of the situation to be interpreted, 
or larger groupings of samples to be inspected as a pattern (thus getting 
more context), and reducing the tendency to semantic oversimplification. 
That is, avoid the tendency to impose a name on the phenomenon without check- 
ing further for data that qualify the name (as by subscripting it) or, 
qualify th*^ interpretation indicating that extrapolation and generalization 
or conclusions based on the data examined would be risky. 

Offsetting the spontaneous behavior tendencies requires either continuously 
active motivation by the operator to do so, or the control of the program- 
ming of work samples to which the operator is exposed, and giving him 
immediate error feedback when he makes mistakes. The latter is done in train- 
ing. It may be difficult in operations unless training activities can be 
interspersed. Thus, what tends to happen is that the operator may become 
fast and reliable in interpreting standard situations, but unreliable for 
interpreting novel circumstances because of his stereotypical behavior. 

Consider the interpretation to ue Lentati^e. In amplification of the fore- 
going point, tne operator should be 'trainer!' (and wtivated to continue after 
training) to treat any interpretation as a tentative hypothesis that can be 
confirmed only to a degree by (a) more data samples, and (b) testing other 
hypotheses against w^dor contexts of the data presented. One is tempted to 
say, "Check the conclusion against common sense." Failure of the conclusion 
to fit "common sense" is not r/jfficient basis for denying it, but should be 
a basis for examining more data if they are available, or for making the 
interpretation ^very tentative. 
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The capacity for this kind of semantic sophistication may involve sub- 
stantial individual differences that may counter-bal ance even lengthy and 
dedicated training. 

Optimize chunk of i nformation for interpretin g. A significant performance 
strategy would aim at optimizing the unit of data that is sampled in 
accordance with the unit of what is being interpreted. Some optimum or at 
least workable unitof pattern should be selected. In language translation, 
this unit would be the phrase, possibly the sentence. Interpretation of 
the intent behind a political maneuver may not have any clear unit of 
pattern structure, so tnat the following may be required. 

Iden tify key variables . Identify key variables and search for data samples 
of these ke^^ variables as a foundation for the interpretation (hypothesis). 
This strategy assumes that interpretations are made in some reasonably well- 
defined situational context. It may be a rather explicit procedural i zati on 
of the steps to be taken in getting data samples to be interpreted. If key 
variables are examined by the operator in a consistent order, he will be 
formatting the activity. To the extent that the format he uses is complete 
(and to the extent that the complete format is maintained in task behavior), 
the chances for valid interpretation are increased. 

--(^strate gic delay in action. A strategic action, whereby the operator de- 
liberately imposes de7ay~betw^en the sampling of data and deriving an 
hypothesis or conclusion from that data, is used to offset a behavioral 
liability .and has performance validity. "Look again before you leap." 
'■Think before you act." It is the pause of critical, rational reflection 
before commitment. The length of the delay may be very brief (e.g., a few 
tenths of a second). Imposing the delay for critical e.xamination is 
probably context-dependent when learned. Thus, the college professor may 
pause when uffered a conclusion about an experiment by a colleague, but 
reflexively respond immediately to a student's remarks that he thinks the 
tests are unfair because How the delay is filled is wliat is important - 
merely counting to ten before responding is beside the point. The probable 
value of the cognitive interval to task performance is that it tends to ' 
avoid spontaneous stereotype response to the stimulus of the immediate 
present, and to substitute a broader reference base of information in 
arriving at and testing the hypothesis or conclusion which is the "inter- 
pretation". This is a learned strategy in coping with input information 
(as It also is with emotional impulses), and is readily unlearned in task 
environments in which unusual patterns . r^^quirincj interpretation are rare. • 

Ua tch f 0 r q u a 1 i f v i n qcu e s . In many task situations, a correct' interpreta- 
tion (as distingUislieJTrom many acts of "recognition" or identification 
which depend on the similarity underlying various cue patterns) depends on 
tne perception of differences between the instance at hand and the reference 
set of instances. (Thus, the statement, "Orientals tend to be more polite 
than Occidentals", may be incorrectly interpreted as "All Orientals are 
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always more polite than any Occidentals at any time." This interpretation 
is a dangerous guide to behavior and is subject to disillusionment and re- 
jection of the correctly s'tated assertion assuming it is actually true 
and a denial of the credibility of the source of the assertion.) The 
operator's failure to detect cues to critical difference(s) in the pattern 
to be interpreted makes progressive improvement in his ability jto interpret 
a class of situations difficult or impossible. . ^ 

It is often necessary for a subordinate to interpret the orders he receives 
from his superiors. Failure to distinguish between the following statements, 
for example, is likely to lead to trouble. "I want you to use your initiative 
but you will be accountable for a high level of judgment." "I want you to 
use your initiative even though it may take a lot of courage to carry out 
the action." The first admonishes daring and implies conservatism and 
caution. The second reverses tliese priorities. 

Each class of task and environment probably has its own sets of key qualifying 
cues which must be learned and applied for the operator to gain interpretive 
expertise. These cues rarel> enter the classroom or instructional syllabus. 
If they are transmitted at all, they are likely to be passed on informally 
from the expert to the novice in the work situation. Such instruction has 
the advantage of direct relevance, but is subject. to liabilities of verbal 
instruction. 

Predictive interpretat ion. Tne purpose of many kinds of interpretation is 
to make a predictiorT T!e"teorology is a technique for interpreting data 
about past and present vveather conditions in orde*" to predict weather in a 
given locale. Interprerinj the behavior of a motorist at an intersection 
is for the purpose of p.-eoicting what he is going to do as it may affect 
your own plans and maneuvers. . Observation of samples of the behavior of a 
maintenance technician who is checking out your aircraft has, as Its 
practical purpose, your prediction as "co whether you will have tro,uble with 
your aircraft in flight. 

Predictions are inherently more unreliable than the interpretation of a 
present state of affa'^^s. It should follow that a broader range of samples ^ 
should be taken and assessed in interpreting the future than interpreting 
the present^ Prediction*' of the future implies a date at which something 
will happen/ trend data must be sampled in order to make the tempora-l pro- 
jection. 

Task strategies need lo bo developed for estimating rates of change and 
projecting points in tii-^e when various events will intersect. When will 
the baseball thv*ov;n by pitcher X be over home plate in batting range? If 
the man up is wel'. instructed, he will have learned to take a systematic 
series of' samples of the ball en route to the plate. ' If the sampling 
series is systematic with respect to standard time intervals, he has an 
experience reference against which to evaluate present data on the ball's 
motion. (For the moment we are disregarding the problem-of the direction 
of the.ball's trajectory as if this were a separate probl em, which psycfio-^ 
logically it probably is not.) If the 'rate is constant, two observations 
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of relative position are enough theoretically to predict its moment of 
arrival, assuming a constant distance between the pitcher and batter and 
fixed distances at which the position of the ball is observed. 

The same principle of time sampling is used by rational managers in assess- 
ing whether a project will be completed by a given date. Of course, manage- 
ment also is aware of the dynamic properties of work and workers oh a given 
-kind of project, and thus can project where changes in relative rate are 
likely to occur. Samples are taken, not randomly, but at certai'n key points 
in progress. These key points are learned through a background of experience. 
The professor in a graduate school can predict whether the student's disserta- 
tion will be completed within a year by considering the. data on which the 
student defines the problefn and the date on which the first datum is collected 

Closely Related Topics 

"Interpret" is closely related to Identify and Decision Making. 
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Choosing a response to fit the situation 

In a broad, inclusive sense, decision making consists 
of organization and processing of the following classes 
of information: need or want variables and situation 
variables (two classes of variables that characterize ^ 
the "problem situation"); response alternatives; and » 
(possibly) some strategy, policy, or rule for matching 
the implications of a response alternative or resource 
available to the profile of needs and situation. 

Choices are usually made in a condition of some uncertainty about actual situa- 
tions, criterion or need, and uncertainty as to how a given choice will work 
out. 

In mechanized decision making, the response alternatives are exclusive of each 
other, and the classes of data and data content are sufficient for operating 
the rule which selects the response from the response repertoire. The rule may 
extend to handling probabilistic data, or even include random, selection devices 
(in order to break action deadlocks). The designer of the rule includes an im- 
plicit definition of "purpose" in the formulation of the rule and principles 
for selecting a response to a situation. 

Purpose is rarely simple in real-life human decision making, where substantial 
commitments are made by. putting a choice into action. Intentions and wants are 
numerous, interactive, and tend to remain in flux. Ordinarily , one must trade 
off ohe kind of objective with another objective, for example, low cost versus 
high quality or dependability. This is a flux situation (like a stewpot) .rather 
than a matrix formulation of values arranged systematically according to magni- 
tudes of X and y arguments. 

Human decisions in real life tend to be characterized by uncertainty about many 
things: what one wants; how much one will pay for what he wants in terms of 
penalties or loss of resource; what the situation really consists of, or even 
what one should know about the situation in order to define "the problem;" the 
range of responses from which choice may be relev-ant; implications of the re- 
sponse in terms of commitment; or probability of success or ensuing entangle- 
ments. The imposition of probability estimating on correctness of interpreta- 
tion what the facts actually are below the pr^esented facade -- or on response 
capabilities (for success or failure) may or may not materially contribute to 
the rationality of the choice-making operation. 

As an example of the interplay of these factors, recall som^ major, non-impulsive 
purchase such as that of an automobile or a dwellin^g. Or, consider a decision to 
overtake and pass an automobile on a wet, unfamiliar two-lane highway with a 
curve looming ahead. 

The actual operation of making a rational choice implies that all of the avail- 
able (and. hypothesized) information is taken into account. In come cases, deci- 
sion situations are formally structured so that the process can be segmented 
and each segment treated serially. The so-called "decision tree" is an example 
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of segmenting a complex decision into units that can be treated one at a time. 
In operational situations, however, the operator"s rational choice tends to be 
limited by what he can hold in mind at one time and, thus, is subject to the 
limits of his working or short-term memory. This limitation implies a psycho- 
logical screening process. The filter in this screen may consist of preferences, 
stereotypes, projections, dramatized cues and wishes — the variety of factors 
which we have examined under the rubric of Interpret. In fact, the definition, 
of the process of interpreting precisely fits the "situation analysis" aspects- 
of decision making. Interpreting is normally a part of decision making. 

The limitations of workihX memory for holding information makes it necessary 
to impose simplifications iV) what is thought about and short cuts in "logical" 
processing. Constructive thinking may consist in large part of the examination 
of a series of facets of the situation and sifting out those factors whi'qh are 
marginal context from those which are central to the choice. One major factor 
in effective skill in complex choice-making may consist of a pattern of learn- 
ing experiences which enable quick separation of marginal from central factors. 

A second factor in decision making skill would consist of the functional in- 
crease of the information which working memory can hold in the process of reach- 
ing a choice. The subsection on decision making task strategies discusses 
hypotheses for ordering the assimilation of information content about a given \ 
situation so as tq, faci li tate its retention in working memory. \ 

As a person works on a job, he tends to separate decisions which, from his point 
of view, have marginal consequences, from those with dramatic consequences. He 
tends to make marginal decisions by impulse, with little or no reflection. The 
impulse may be ft)unded on an extended series of decisions and consequences in 
h.is experience, so that, in effect, he "recognizes" the proper choice from the 
data presented or inferred. Or he may become adept at mental coin-flipping be- 
cause he does not expect that further thinking or further information (if avail- 
able) is likely to improve the quality of the decision he can make in that situ- 
ation. (This assertion assumes the operator is not clinically disturbed.) 

With practice and a subjectively reasonable batting average, the ^operator tends 
to develop format characteristics for classes of decision that continue to re- 
quire reflection. The format is much like an implicit, standardized question- 
naire of conceptualized variables. If possible, the decision maker structures 
his Jnquiry so that the questions are answered in a standard, habitual sequence. 
The ordering of the content may be based on the sequential ordering of data 
variables from the information source, or the ordering may be based on the 
ability of the decision maker to control the sequence of inquiry. The exper- 
ienced decision maker also tends to structure his inquiry by acquired criteria 
of relevance. (Relevance is determined by the answer to a question: Does 
this kind of information contribute to the selection of a response I am able to 
make?) This filtering process obviously simplifies the amount of information 
which the decision maker must process. Although psychologically desirable, this 
tendency can, on occasion, lead to errors in. decision making that are serious 
from an operational standpoint. 

Just as many repetitions of the interpretive process in a class of situations 
can result in reducing the processing activity to an "identification" of an 
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individual "member" in that class, so too can the repetition of deci sion ^making 
activities (originally requiring both interpretation and response selection 
through^ vicarious .testing of response implications) lead to virtually procedural 
response at the near automatic level of behavior. This process is behaviorally 
adaptive according to the processing economy of the human as an adaptive system. 
On occasion, Internal adaptation of this sort has operational liabilities. 
Where such liabilities can be anticipated, the training process should attempt 
to inhibit the progress of the reflective process towards mere automatic re-' 
sponse. But arresting this development has several penalties: longer delays 
between the psychological onset of the problem situation and the action response 
(thinking takes^time); diminished ability to attend to other things going on at 
the same time (thinking us,es up channel capacity). 

Most of these factors are itemized in the following outline of behavior ten- 
dencies. 

Behavior Tendencies , • 

Short cuts can be expected in sampling and processing all classes of information 
cited in the definition with which we began this section on decision making. 

Standardization of goal variables . Need variables or goal variables become 
standardized and formalized and may eventually be dropped from consideration 
when the decision operation Is reduced to a procedure or habitual formula. 
Short-term goals tend to predominate over longer term goals. Some goal vari- 
ables may be given inordinate priority over others, despite unhappy expertences 
' arising from this neglect (e.g. , deciding to have those additional drinks despite 
the inevitability of tomorrow's hangover or tonight's traffic hazards). 

Situation variables . Restricted sampling of the situation variables tends to 
occur as decisions in the subject domain become habitual and the internal decision 
format is stabilized, 

Response al ternati ves . There tends to be restricted testing of the implications 
of response alternatives, and Umited searching for new ones. There is a ten- 
' dency to over-generalize successful and unsuccessful responses , 'depending on 
prevail ing mood. 

Rationalizing choice policy . The operator tends to rationalize that a policy 
or principle has been followed a^fter a choice has been made, rather than as a 
rationale and guide to choice selection. 

Formatted structure . Repetitive decision making as part of a job dealing with' 
more or similar classes of situations tends to become formatted. The same set 
of variables is examined In the same sequence. Fewer variables in the problem 
are considered, unless the decision maker works from an objective checklist 
that requires him to fill in data in each category. Even the use of a checklist 
can become a ritual in which only a restricted set of data is treated as rele- 
vant. The range and variety of response choices tends to shrink radically to 
a few prepotent favorites. (Observers may call such a decision maker "preju^- 
diced.") The cognitive operations of weighing and thinking through the Impli- 
cations of situation, needs, and alternatives, tend to be reduced and the choice 
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tends to be elicited 



4. J, automatically. These outcomes may be positively" adaotivp 

to pe,-formance demands if a limited set of patterns is frequently repeated and 
changf'^""^^ repertoire and effectiveness of respective responses does not 



In many cases, decisions must be made about events taking place in the future 
which means there s some degree of uncertainty regarding conditions, priority 
?Lw n ' jvailabi ity of resources to implement a choice. A conservative stra- 
tegy or policy will hedge on connitment or lean toward choices that have tradi- 
tionally been successful and never disastrous. A more daring policy may pro- 
ject optimism into the uncertainty and commit more dangerously: 

Behavioral strategy in dealing wi th . uncertainty is a function of individual 
personality variables, mood, and, of course, the interpretation of 
of experiences. 

Task Strategies 



a pattern 



Task strategies proposed in this section are restricted to those intended to 
reduce the liabilities and limitations of general behavioral tendencies in de- 
cision making - where these tendencies do constitute liabilities. The propo- 
sals will, therefore, seem general rather than specific. The application of 
some may seem inconsistent with others, but this is the nature of the tradeoffs 
compromises,, and qualifications which make up a complex human s'kill. 

The central idea of the .following task strategies is the structuring (not re- 
duction) .of a class of decisions in a job to' a pro_ceciur_e. This is not neces- ' 
san.iy equivalent to making the decision process ll fogeiiher mechanical, although 
where Practicable, it is likely to be desirable. The aspects of the procedure 
til TfTirfn/f ''''ll^^u^iy ' combination of task design and training regimen, 
the difficulties which the novice has in a class of complex decision'situations 



In organizational practice, this is commonplace, 
pared for decis ion-fiiakihg situations encountered 
operations in ah office. Consider an admissions office 
cisions are made to accept or not accept applicants. A 
is designed to cull information about the applicant on a set 
ables._ Questionnaire data may be supplemented by i ntervievv data ; the 
interviewer tends to work from an implicit or 



which are an extension of the 



"A format is eventually pre- 
many. times,, perhaps as routine 
in a college where de- 
standard questionnaire 
of relevant vari-^ 
trai ned 

explicit check list of factors 



, ,y J ^. questionnaire. The staff also develops a more 

or less well-defined set of requi remen lb for admission. These requirements of 
course, may te modified by circumstances. The criterion variables 
ance should approximate the variables in the questionnaire which 
applicant. 



for accept- 
describe the 



Thus far, the collection of data relevant to the decision has become "well- 
structured ' As a result of historical data and experience, "standard opera- 
ting procedures" can be propounded. The data describing the case can be 
ordered in a sequence to match the "model" of acceptance or of rejection, there- 
by reducing what the human decision maker has to bear in mind at one time. He 
stores It in a "message format". > 
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' The process for reachiing a choice may now beg-^n. Factors in^the situation that 
match equivalent factors in the acceptance qiodel can^ henceforth, be generally 
ignored. The decision policy may make rejection,.(an answer of "no'') automatic 
if some subset of negative factors is not matched. (Females, for example, used 
to be^au'tornatically rejected if they applied to a boys scho,ol.) Some factor(s)- 
^ niay be given priority and outweigh some deficiency; e.g., athletic ability may 
' be given priority over the applicant's overall academic standing. A high value 
in one desirable variable is traded against a low value in another desirable 



Clearly, although this process is not mechanized, it can -become well -organized 
through task. design. When a mass of information becomes organized, the deci-.. 
siQn maker can better cope with a mul ti -variate problem, and better approxi- 
mate a rational model even with semi-rational or non-rational information such 
as beliefs., va.lues, -and preferen^ces, as well as sutJjective estimates of proba- 
bility. <^ , 

Operations research it a technical example of the attempt to formalize decision 
- structure^ .but with the addition of metrics, quantitative assumptions, and ex- 
plicit policy rules for optimizing criterion variables. According to critics, 
practical difficulties with it in some complex situations have arisen from in- 
appropriate assumptions and a reluctance to allow the human to interact intelli- 
gently with the problem. Other severe problems arise from -having -to treat a 
single variable of purpose or utility at a" time, and/or from questionable assump- 
tions in converting multiple purposes into a single "utility" scale. 

The following kinds of format operations are intended to facilitate the* learning 
of decision making in a given task envi rorjnient. All issues are simplified, in 
many cases oveirsimpl i f ied. 



Establish the range of r espon se opti ons. Determination of the irange of responses 
from which a particular decision inaRor may ptioperly select is one facet of de- 
frnUiy the man's job. It is the bounding of lines of authority. In some cases, 
as in the case of the'cpllege admissions officer,^ this may be relatively easy to. 
do; in other cases, as in the case of the leader of an exploratory platoon, such 
boundaries ma^ be practically impossible to predict. Where a response qlass can 
be meaningfully specified, criteria of what information is, relevant and what is 
not, become established. Information in a givon decision, si tuation is relevant 
if it contributes to choosing,^ a. correct or adequate response from* the permissible 
response set.. . * . 

The liability^in limiting the permissible response set-is that the decision maker 
is deterred from innovative behavior (e.g. , inventing unanticipated new alterna- 
tive?) which m^^y be more operationally adaptive and useful than any already con- 
sidered. This is an example of .the usual tradeoff between reliability or pffi-.. 
ciency versus flexibility. ' 

StV^ucture of hierar c hy of de cisions. Some complex decisions may be formatted in- 
to a sequential set ot subde'cusions". "If decision A is yes, go on to B; if A is 
no, consider X. If B is yes, go on to consider C." The decision to take a job 
with a new employer might be sequenced. as fe^llows:. Is the pay likely to .be * c 



variable. 
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acceptable, ai least as a minimum? If the answer is "no," there is little point 
to proceed with inquiry because the final decision will be to reject the job 
offer. Assuming a "yes" answer, the next question and decision may be; Can I 
dp the work? Rarely can an unqualified "Yes" or "No" decision be made, but this 
is not always necessary. Rather, if the decision maker must make many decisions 
of this kind, jie will be better off if he attempts to sequence the issues in a 
standardized order. One outcome of his experiences will be a consistent refer- 
ence pattern. 

The decision tree paradigm is a formal example of constructing a hierarchy of 
decisions. It includes the additional variables of payoff and probability of 
given "states of nature." ,^ 

Develop formats for acquiring an d remembering situation data . A class of 
choice-making situation may be described in as few as two variables or as many 
as a score or more, The same variables appear in each occasion of this kind of 
decision, although a different value, magnitude, or content is likely. Left to 
his own experiences in making decisions, the dedsi^on maker spontaneously devel- 
ops an ordered list of inquiry variables which he addresses to a given situation, 
Training that is directed to the formatting and sequencing of these situational 
variables can accelerate "the learning process . The student learns more quickly 
how to acquire and hold in mind the data describing the specific situation con- 
fronting him. As a desirable by-product, the ability to hold this information 
m'ore readily in mind gives more mental "work-space" for thinking towards a de- 
cision. It also provides work-space to inquire about and hold in mind contin- 
gencies that do not fit the standard inquiry format. In layman's language, this 
.is called "sizing up the situation." 

An exaftiple of this dove lopr.ient is in order, A military policeman patrolling a 
city makes decisions as -to whether to ignore a "troubie" situation involving 
military personnel, intercede with verbal advice or orders, have the offender 
picked up and returned to. camp without arrest, or formally arrest and charge 
the- offender. He may examine the situation for data responsive to the following 
criteria: Is he damaging prestige to the uniform and service? Is the person 
creating or likely to create a progressively worse disturbance? Is he damaging 
persons or property or likely to do so?' Is he breaking a civil or military 
taw? ;>Should he be apprehended for his own good or protection? Is the attention 
and-resource necessary to intervene in this case justified compared to potential 
need to address more^serious problems that may arise elsewhere? The appearance 
of indecision in the military policeman is a liability to his exercise of auth- 
ority, He should make up his mind quickly. The mure quickly a complete format 
of i7iquiry can be applied, the more quickly he can decide. If he has learned 
to apply criteria automatically in a given sequence, he can apply them more 
quickly and more re\|iably. This is the rationale for serializing the format 
for inquiring about the status of a situation. 

Getting situational dat a. The business executive trying to make up his mind 
about a proposed merger and the would-be vacationer urying to make up his mind 
whether to commit a down payment for Resort A or Resort B, complain about not 
having ''enough data" to make a- reasonable decision. Where choices have only 
probabilistic outcomes, there never is "enough" data for certainty.. 
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Enough data for making a decision has at least three kinds of meaning. One has 
to do with the subjective sense of assurance that a given choice of action is 
better than known alternatives. On. this, there is some evidence that there are 
large individual differences. The so-called intuitives require relatively 
vlittle data formaking an intuitive leap to a choice; others continue to be un- 
certain even wi'th logically complete information for making a choice./ If this 
is, indeed, a fairly cross-sectional characteristic of personality, selection 
procedures of one kind or another seem desirable for the decision maker in a 
given kind of environment. 

Enough data also has meaning in a rational context. If a conceptual model of 
situational -variables in a class of decision, such as initiating a/'battle cam- 
paign or making a merger of businesses, has been developed, then absence of 
data about one or more of these variables is, indeed, inadequate data for making 
a choice. In such a case, two principles apply. One principle is that there 
usually are a few central variables whi^h, having given values , /should lead to 
peremptory rejection or acceptance of an hypothesis for the outcome of a given 
action. Data on these variables are likely to be worth a high price compared 
to data about other variables. The principle of selectivity applies. Put more 
effort into getting key data for accepting or rejecting a tentative choice. 
The popular literature about leadership emphasizes the ability to seize on these 
1<ey variables and to deal excl usi vely ;wi th them. It is not certain that diff- 
erent individuals operate well with different key variables 1n a class of deci- 
sions, or that key vefriables can be abstracted for generalized training. The 
latter hypothesis is worth inquiry where instruction about important decisions 
is prepared. 

A second principle In gathering information for decisions is, in effect, the 
law of diminishing, returns. After ^ome minimum level of information, the amount 
of additional useful information depreases for each unit of additional cost. 
For his own choice-making purposes, the seasoned decision (iiaker probably learns 
subjectively when the inflection point has been reached. 

A third principle for information gathering also applies./ Much less information 
is generally required to reject an Alternative that to aq'cept it. An alterna- 
tive may be rejected because 1t faiT\s to meet some one or more necessary condi- 
tions. Ordinarily, an alternative i^^ reasonably accepted only when it fulfills 
sufficient conditions for acceptance.v Generally, sufficient conditions consist 
of more than one necessary condi tion.\. If there are few alternatives, the most 
attractive is that which remains after the others have been rejected on the 
basis of failing to meet a necessary condition. For example, a secretary must 
be/ assigned to a job which requires nominal typing and ^dictation speeds, but 
requires fluency in the Greek language. \ In getting data about American candi- 
dates for the job, the most efficient first question to ask would be: ' Do you 
speak Greek? \ ^ _ 

where a type of decision is recurrent, either for an individual or an organiza- 
tion, formats for the collection and organization of situational data become 
standardized. Standardized formats have advantages for efficiency, and occa- 
sional penalties of inflexibility. An instfuctional milieu may assist the 
decision maker in helping him to test the effectiveness and efficiency of the 
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existiny data formats ht' uses, and in proposiny others that may be more adequate 
becWe (a) they are more complete, (b) they are sliorter and more efficient, or 
(c) tn^ are up-to-date. 

Astute bargainers know that tiiin'ny is important -- the time at which new infor- 
mation is'introduced into a decision may have influence disproportionate to its 
value if introduced earlier. Thus., a cancer patient may be told that he can 
live for perhaps six months without surgery, but with surgery he has a fifty- 
fifty chance of dying in the operating room. The patient vacillates. After an 
extended period of indecision, the physician may tell him that his chances are 
closer to 60-40 in favor of survival after suraery. He may then be more likely 
to choose surgery. If, initially, he had been^told that his chances were 60-40 
in favor of survival, the effect might have been about the same as being told 
his chances were 50-50 -- indecision-. If a genuine deadlock among alternatives 
has been reached, anything, no matter how- minor, which tips the scale one way 
or the other, is logically justified in swaying the decision. Decisions are 
rarely reached by such preci se 'weighing of factors. 

ijr ocessinqJ ^h^JjT^fo^rinatio n in making a ch oice. The thinking process for reach- 
ing a choice after reflection is likely to~Fe recursive. One set of factors 
after another is weighed. One liability of recursive action is the likelihood 
of circular activity which leads to no conclusion at all, and where, in final 
desperation, the choice is based on a kind of coin-flipping. 

Another liability is that because of information overload, a solution is sought 
by abandorting the original criteria for choice, and selecting some criterion 
that has a particularly attractive value which is highly visible in one of the 
alternatives. Thus, one approaches the purchase of an automobile with multiple 
criteria -- estimates of durability, economy, serviceability, versatility in • 
function, attractiveness and the choice is finally based solely on which car 
looks most attractive. 

A strategy for coping with either or both of these liabilities is to firmly re- 
ject a candidate alternative when the application of rational criteria has 
shown that it should be rejected. This means, In effect, put further thought 
about this alternative out of mind. By definition, the sophisticated decision 
maker is able to do this. 

In human affairs, the final act of decision is often an intuitive action., (Call- 
ing it intuitive is a confession of ignorance, about the process.) After a deci- 
sion has been announced, many persons feel obliged to make their decision appear 
rational, to rationalize, The credibility of explanations of the process for' 
reaching a conclusion may be dubious, although statements about ' factors that in- 
fluenced or were decisive in shaping a choice may be valid. 

The reader is reminded that the kind of decisions we have been discussing do not 
have strictly deductive or cal cul ational procedures for producing answers which 
then are accepted as given. 

If reaching a positive decision to act is preceded by treating the impending 
choice as an hypothesis, remote as well as immediate consequences of the choice 
should be conjectured. This is best done by imagining the train of consequences 
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in as concrete terms as possible. If the decision will affect the interests 
and actions of a group of subordinates, consider the effects on various indi- 
vidual members of the group. It is easier for a preferred action to be ration- ' 
alized by thinking qf abstractions (which invite stereotypes) than by samples 
of reality. This is likely to holdto some degree even when the "samples of 
reality" occur in the imagination. 

Exploratory testing of partial decision s. A strategy used by cautious decision 
makers is to commit only tentatively to a course of action, gaining information 
about the situation relevant to further commitment or withdrawal from the con- 
sequences of tentative action. This is probably a widespread practice in our 
daily life as well as in institutional life. Its liabilities are obvious. The 
sample data obtained from a temporary and partial commitment may not be repre- 
sentative of response to a full coinmitment extended over time. If the explora- 
tory outcome is negative, a resource may have been needlessly lost. If the out- 
come is positive, the crucial test may still remain. 

.These assertions must be qualified. Where an entire plan can be put into effect, 
but restricted to representative samples of the population to be affected, the 

■advantages of exploratory or pilot testing are clear. It differs from the pro- 
cedure of taking only one step at a time in the equivalent of a campaign, in 
order to determine the probable effectiveness of the campaign. 

Modeling, simulation, and gaming exercises fill the role of exploratory testing- 
of an hypothesis, where time and resource permit. These procedures are exten- 
sively described in their respective literatures, and are not considered appl i - 
cable to our present purposes. What is germane is the decision by one or more 
people whether or not to put intb practice an hypothesis after these data have 
been collected. A judgment has to be made on the extent to which these models 
and the data derived from them are sufficiently valid for committing tc a choice 
in the real world. 

Comment 

In this section, we have not dealt with formal decision-making strategy represented 
by such disciplines as game theory, operations research, and simulation. If these 
disciplines are regarded as data-generating tools (rather than as automated deci- 
sionmakers), they provide input to an individual who chooses to be guided or not 
guided by their output. There is an extensive literature on human behavior in 
choice-making situations. One excellent analytic summary of. this literature was 
prepared by Schum (1970)*. The extent of the relevance of much of this litera- 
ture, based on experimental conditions having well -structured problems (e.g., 
gambling-type games), is open to discussion, and perhaps subject to the inter- 
pretive skill of the reader who is seeking behavioral generalizations. 



IT 

In DeGreene, K.B., Editor, 1970. Systems Psychology. N.Y. McGraw-Hill. 
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Matching resources over time to expectations of needs 

Rules and facilities for predicting what future 
sets of conditions will occur, what responses to 
make to them, and in what order, so that given 
goals are likely to be achieved. 

Planning Operations 



0 Predicting the future — using historical and present information to 
anticipate which of a set of alternative states will occur at some 
future date or time interval. 

© Exercising priority rules for determining which of several anticipated 
states to give priority of attention. 

0 Determining the set of response capabilities required for effective 
response to the expected condition or state. 

0 Scheduling the resource In making the response so that the resource 
is available when the expected condition occurs. 

The planner combines the functions of predictor, resource selector, and resource 
scheduler. 



Discussion • 

Planning is a form of decision making. But planning consists of constructing an 
action ^pattern based on the outcome of a number of choices, rather than merely 
selecting from among choices. Since a plan deals with the future and the future 
is never absolutely predictable, planning involves uncertainty, hence, choosing 
from among various probabilities. Inevitably, there are planning alternatives 
with respect to goal commitment, goal routes*, and resource selection and schedul- 
ing. Instruction in planning must provide explicit awareness of these fundamen- 
tal variables in the planning operation, or it will be inefficient except for 
prescribed, restricted, and highly formalized "planning". 

A planning job or task performed by an individual is usually restricted to an 
information domain; the planning task deals with a restricted set of variables 
in terms of goals, relevant environmental states, and relevant resources. Eco- 
nomic planners deal with cost, revenue, ^nd profit variables; inventory planners 
deal with goods flow and stdj^age variables; aircraft mission planners deal with' 
the variables of payload, distance, targets, routes, equipment, personnel . 
There are short-term planners and long-term planners, and each works in a some- 
what different domain of variables (including goal variables). Any planner's 
job deals with a relatively defined and restricted context of information. 

The planner, like any other decision maker, tends to learn or to create an impli- ' 
cit structure or "model" of the variables associated with his areas of responsi- 
bility. In some cases, this model may become as explicit as a checklist. 
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Anticipation of requirements can be standardized if, in every planning cycle, 
the requirements tend to be about the same. Allocation- of resources tends to 
be about the same if, on every planning cycle, the pool of resources is the 
same, and all priorities are about the same. Under such conditions, planning 
becomes the operation of a set of rules and fonnats. 

Since planning is a special form of decision making, the reader should refer to 
_the entire subsection on Decide/Select. The material in that discussion is not 
repeated here. 

Behavior Tendencies . . . 

See Behavior Tendencies described, under Decide/Select. 
Task Strategies 

The Task Strategies cited under Decide/Select should be referred to, along with 
the fol lowing. 

Critical path schematics . The concept underlying PERT (Program Eval uation 'Re- 
view Technique; and critical path methods is the fundamental structure for any 
kind of planning operation. In summary, this consists of (a) constructing a 
scenario of the major points along the route towards completion o.f an objective, 
and (b) determining, for each of these major points, necessity for. desirable 
resources for proceeding into the next phase of activity towards* the objective. 
It is, in effect, a sequence chart of parallel and converging actions. , A major 
point may be defined as a position in a path where a new pattern of resources 
is required, or some condition or state must be achieved. 

Consider a plan for a business trip to another city. Planning is required for: 
getting off on time, transportation to the airport, an airplane ticket to the 
destination, transportation from the destination airport to the office to be 
visited, arrangements for meeting the desired people there at an expected time, 
transportation back to the airport, return passage, and so forth. This scenario 
suggests the elements to be planned for; i.e., transportation to airport, air- 
plane reservation, and so forth. \ * 

Naive planners expect transportation to be ready when cal led, \airplanes to.be 
strictly on time, hotel reservations always to be in order. wKere success is 
extremely important, good planning anticipates contingencies .ancl, prepares some 
fallback for them. Planning for contingencies inevitably calls )ypr redundant 
or additional resource(s) to be available. Judgment (based on a aombination 
of knowledge and iinagi nation) estimates the chances of given thing^ going wrong, 
the chances of an impromptu remedy being available, and balances the,se against 
the cost of providing a fallback resource. \^ 

One class of resource can often be balanced against another. If you arfe concerned 
about having a reservation for a given flight at 10 o'clock in the morning.^ you 
might (if unethical) arrange booking for an al ternate ^f light at about 10 oVlock 
to .the same destination on anot;her airline. This could cost dollars or reputa-r 
tion. If you had ample. time resource, and were willing to wait until the 11 
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-o'clock flight m case the one booked at 10 o'clock did not work out, this would 
be a f::llback. 

o 

Strategic planning -- takipg into account contingencies seeks for resources 
that are least costly in terms of the organization's resource and most versa- 
tile in terms of potential mission contingencies. In 'ivilian and garrison 
operations, money is a highly versatile resource, but i l has little value when 
a carrier-based aircraft is running out of gas one hundred miles from the deck. 

The tighter the plan in terms of permissible" resource to be committed "(especially 
timej, the more vulnerable is success to the occurrence of the unexpected This 
IS why a vast amount of planning went into thinking through, in advance, the 
variety of contingencies that could occur in manned -space missions. Every mission 
has had one or more contingencies. Incidentally, one should not forget the human 
as a planned resource to be treated as a range and degree of competence whose 
costs include selection, training, and maintenance. 

In summary, a dimension of strategic planning consists of creating a schematic 
or scenario of the flow of events in a mission and of checkpoints where a new 
class of resources is required. That .schematic should suggest what resources 
are needed, -and when, and should also point to the possible contingencies re- 
quiri'ng fallback resource. 

Contingency resources. Here, we c6ntinue a theme from the previous discussion. 
A statement in quantitative terms is proposed which is based on a variety of 
experiences and recommendati.ons from both formal and informal planners. After 
careful planning and resource budgeting, allow approximately 20 percent more of 
the b-udgeted resource for contingencies, if the mission occurs on a well-known 
route with well-known resources. But, if the route is unfamiliar or uncertain, 
and/or the resource capabi li ty. i s not well known under the mission conditions, 
allow substantia;i.ly more than a 20 percent reserve in budgeted resource. 

These may be a "pseudo-quantification except where dollars, time, or materials are 
concerned. The actual values may be challenged, and if there is a bettpr empiri- 
cal background for selecting a different value for a class of planning operations 
the better value should be used as a training and operating guide. 

Training Im plications 

Few duties and responsibilities do not entail some planning. Formal curricula 
would profitably develop conceptual and practice content in the art and disci- 
pline of, planning in the context of the given duty and responsibility In 
daily life, at work or in private, the fact is that few individuals seem to be 
very good planners. Planning is not a skill that is necessarily picked up 
spontanejsusly through on-the-job experiences. 

At the last, training might provide examples of structural planning formats for ' 
various relevant job contexts. The "format" is not merely a printed form, but 
also includes some exercises in constructive thinking 
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•Holding something temporarily 

Rules and facilities for holding messages or 
parts of messages in temporary storage for use 
at later times during a task or duty cycle, or 
for combining with other information during 
the cycle. 

The above is a convenient definition for short-term memory, perhaps more appro- 
priately called "working memory." 

In general, operations are equivalent to those described for Recall Enabling 
Information (Long-Term Mefr.ory) . 

Examples 

• A typist reading a sentence and holding it in mind while her fingers 
key the symbols. . c 

Of A computer r^egister holding data presently to be combined with other 
data, in an' operation . 

• The retention of symbo.ls or messages in a buffering device for trans- 
lating fnto a different transmittal rate or frequency. 

• A decision maker holding in mind the objectives, problem information, 
and choice-making policies in the co.urse of reaching a decision. 

• A troubl eshooter keeping in mind the tests he has already made and 
his inferences from those tests, while selecting the next test. 



Factors 



ERLC 



The greater the number of input charinels, ^variables, codes, and chunks 
of information input that must be integrated over time to reach a de- 
cision or select a response pattern, the larger the short-term storage 
capabili,ty required. Human s«hort-term memory is limited (according to 
a variety of factors) but can be functionally increased by practice of 
context and by^ regularizing or formatting the inpu-t, by mnemonic aids, 
and by map-like job aids. 

The greater the variations among n)essage sizes and message rates of 
transmitted and processing data,- the more important the design of 
short-term storage facilities is to the efficiency of the total sys- . 
tern. The greater the kinds and ranges of random variability in 
messages, message sources, interruptions, and noise, and the greater 
the variety and complexity of response pattern alternatives, the 
greater the burden on short-term memory. 

Information elements in short-tenu storage must be functionally address 
able as and *when relevant to the decision to be reached or the problem 
to be solved. Some address codes, or mnemonics, are more efficient^ 
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than others in a given system. A given "system" in this context is 
an individual who is likely to differ from others in his mnemonic 
. preferences, . 

Discussion 

The complexity and variety of short-term memory phenomena, and the central impor- 
tance of this function to tasks that have cognitive or perceptual . content, make 
short-term memory the most fruitful of all human functions for strategy design. 
In fact, practically every function in the present set has direct or indirect 
bearing on short-term memory, and many of the strategies recommended for other 
functions might as readily have been cited here under short-term memory. 

Individual differences in cognitive ability should nowhere show themselves more 
significantly than in short-term memory, except perhaps for long-term memory. 
This would seem a truism. Clearly, the complexity and scope of thinking and 
problem solving seems to consist largely in the kind and amount of information 
an individual can hold in mind at one time. The ability of some individuals to 
hold in mind long strings of mathematical code, for example, while others can 
hold in mind large amounts of information in the form of a map, while still 
others can hold the representation of an intricate model or processing opera- 
tions in mind and the abiTity in each case to "think" with that content -- 
clearly implies aTTeast some of the vast^differences in our individual mental 
capabilities. 

We have individual "preferences" for the kind of code or cognitive representa- 
tion of data presented to us for thinking purposes in short- term retention-. 
Experienced tutors learn that one way of representing a concept or pattern of 
relations and operations to a student may be almost completely useless to his 
comprehension, whereas another symbolic structure of the same content may quickly 
communicate the ideas. 

Individual differences in the kinds of code that can be readily apprehended and 
retained in short-'tenn memory partly explain the emergence of different patterns 
of performance ability among individuals at different stages of learning. At 
early stages of learning, the ability to remember and translate verbal instruc- 
tions regarding "w„hat to do next" may b& the major differentiator among rapid 
and slow learners. But at higher levels of task learning, when these self- 
instructing mediators tend to drop out, the code patterns and "conceptual models" 
of the task content itself dominate performance levels. A person who is poor 
at remembering strings of verb'al instructions in learning how to go about .reading 
a map, may be very good indeed at remembering the map content itself, once he' 
has mastered the procedural operations for inspecting maps for a given task pur- 
pose. ' . 

One individual may be quick to learn the verbal self-instructional content of 
a set of manual procedures, but be relatively poor at later stages of learning 
that depend on muscle coordination and perhaps on the ability to code patterns 
of perceptual information and transform it "spontaneously" (rather than cogni- 
tively) into patterns of muscle activity. 
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The implications for instructional technique of knowledge about individual diff- 
erences in short-term "codi.ng" ..preferences and abilities are almost certain to 
be more important to training technology than the practice of letting each stu- 
dent practice from the same content at his individual learning rate, 

..The content of short-term is not necessarily limited to what the operator can 
tell about or to what he is aware of. Short-term memory seems to have sublimi- 
nal fringes. One may be aware of not having completed some task action without 
remembering its content. While reading a page of text, an individual may be 
interrupted. When he returns to reading the text, his eye tends, spontaneously, 
to find the location on. the page where the interruption occurred. He spontane- 
ously begins reading where he stopped at the interruption. The place where "the 
eye" was interrupted must have 'been stored in short-term memory. 

Short-tenm memory may well be the basis for the organizing of data that results 
in- the selection of a "set to respond,!' Data organization may be quite complex, 
as when one speaks and listens in French to the dinner companion on the left 
and in Chinese .to the guest on the right. In sight reading of music from a 
score, the key signature indicates the pattern of notes to be struck when a 
given symbol for a note is shown on the staff. This key signature invokes a 
very specific repertoire set. But sometimes an "accidental" occurs a 
symbol in front of a note that countermands the signature. The key signature 
may indicate that a note should be flatted, which the accidental symbol says 
should be played ''natural." The accidental value for the note applies through- 
out the measure, and the musician must remember to play the same note with the 
accidental value whenever it reappears in that .measure; in the next measure,' 
the accidental value no longer applies. This is an example of a set within a 
set whicFi must be managed within short-term memory. 

It is not -necessary to imply that the code to an entire response repertoire 
making up 'a "set to respond to a context" must be held in short-term memory 
work space if indeed there is such a "space." But it is necessary to postu- 
late some kind of code representation in short- tenn nienio ry which selectively 
accesses that bounded repertoire. . 

In short-term memory the structure for a command to search long-tenn -memory is' 
created. We are now speaking of a voluntary search into deep memory. "How in 
the past did I remove a gasket from an engine block without tearing the gasket?" 
Individuals differ in the format and code that is congenial in long-term memory, 
just as they do in short-term memory -- and probably in quite parallel ways. 
From- our personal history of successes and fa/i lures in memory searches, we know 
that starting with one kind of^picture or pattern seems to have, better chances 
of uncovering the information we seek than do ofher patterns. (The subsection 
on Recall Enabling Information (Long-Term Memory) contains an extended discussion 
of this issue.) To the extent that education and training are not concerned 
merely with getting students to pass tests, they could be directed to the design 
of what logicians call "search arguments" in. the psychological sense. The con- 
struction of a search argument is the equivalent of a strategy, used deliberately 
by the student for searching "what he knows." Unless the search picture (or 
argument) can effectively be held in short-temi memory 'as a single piece or 
pattern, the capacity for search is limited. 
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In an associative system (in the stimulus-stimulus sense rather than -thP .tim,, 
Jus-response sense) when a "response" is made, it eliciLX nred^Jnn.it^^n J" 
make other responses with which'it has assoc at ve nks As ?hes ?e no s . 
occur (implicUly and subliminally or explicitly with awarenLsT they Senerate 
an expanding web of stilj other responses. The fact of the occu^ren'e of ?he 
response, even sub iminally, strengthens the predisposition ?oJhe entire re- 
sponse family in the web. It is like initiating a reverberation into a self ' 

code 0? resnon es Predisposition to respond with a large vocabulary 

coae OT responses - that takes time. This way of conceptualizing the oroblem 

tical to each other, at- least from an operational standpoint 

These general comments about short-tem memory may serve less as direct tutnri;,! 
• ^.n]rT"' f^^°^\d^^e^ that generate short-term LnoTy sir Lg es than ?o i 
?h^P IL^^'^'"^ ^^'^ short-term memory phenomena present rich opportun ty fi? 
the development and application of behavioral strategies, both in the shaoe nf 
human engineering job supports and in instructional technique ^ ° 

gehavioral Tendenci es_ 

wit!! th°at''?a'k ^?'r^iil°%\^''.^' '""^ processing of information associated 
witn tnat task, a number of tendencies can be expected. • 

Reduction of content. The amount of information that passes throuah coanitivp 

TiclTlVfl'"'' ^° ^^'"^ ^^^^^ established br^he i onuo n p a - 

tices of the operator for feedback sampling. That is^ progressive y less "mean 
ingfulness 15 projected by the operator from long-tem Sry onto^sta darS 

standardized formats. »re or less standardized formats or matrices for the 
m ?s ; r UsII^ Of information develop. An example of someS t (e f r- 
??^n I^H . complex non-repetitive dependencies between iUut informa'- 

f n majo n5'orstab]e 'T'''' ""^'''^ P™tesses ™agery 

IS one major kind of stable memory reference. The implicit comolex imaap haq 

■ oneS:ne?aJ?nft is^-""'^'"."' !"^"'- '^""^ innate^m^cl^anfr e m " a 
DilSt'fimJe of 5je 1""^ selectively updating it.) An example is the 

o?^a::nyn%-:hent'?rmak^1njlretl°f-rdL^Srf;Hit^ 

^'^^x J^ciS?j^:a\i„f rtti? fz^---- 
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Recall Task-Cycle Information 

information sampled at one time. Tou large a clump for a giver^ level of skill 
results in a reduction of performance effectiveness. Size of clump isalso in- 
fluenced by time pressures from competing demands for attention, and by its con- 
formity to individual level of subjective comfort. Unless modified by deliber- 
ate Operator intent, or by specialized training, the size of clump stabilizes 
at so\ne size less than that of the capability of the operator,, and quije poss- * 
ibly less than optimum for' the task.. This tends to happen among many readers, 
of straight text at "least according to the proponents of hl^h speed reading 
for th^ general population. 

J"ask St<?ategies 

Anticipa\ory sampling . Where the task imposes time dependent relationships be- 
tween inp\ut and output, one objective is to sample from information sufficiently 
in advance of the time that effector response needs to be made to that informa- 
tion, so that the effector response can become organized. Sampling aheac^ can 
provide time for checking supplementary channels of information. Most import- 7' 
ant, the sampling ahead of effector response enables maintenance of a relatively 
even rate of effector processing flow -- and avoids working by fits and starts. 
The training strategy , IS to force the operator to scan ahead of his effector 
activity, after some degree of automaticity and organization of effector acti- 
vities has bken acquired, or more or less coincidental with that stage of learn- 
ing. \ 

Conceptual structure . Where relatively large amounts of variable information 
must oe held in short-term storage, suitable retention of that material is re- - 
quired in a farm that may be independent of the sequence in which it is converted 
into effector'response/ The task strategy is to develop a conceptual "model" 
'Which may inclu'de the support of imagery (of various kinds including visual, 
auditory, kinesthetic) which serves as an updatable reference. Where possible, 
the image should partake of the goal image content and should be organized vyith 
similar content.. The image may be supported by a matrix of verbally specifi- 
able variables or relevance to sampling and to effector execlition. Ideally, 
the reference image is comprehensive enough to provide a single reference for 
all, or most, of the most critical channels of information. A mental picture 
of the attitude of an airplane in reference to the horizon entraces the vari- 
ables of roll, pitch, and yaw; .information about each of these variables may 
be transmitted through a separate display channel. 

Short and l o ng-terni memory code . The content and code of short-term memory 
should be consistent with the code and "indexing" of information tn long-term . 
memory, so that retrievalof information or "programs" from long-term memory 
is fast and reliable. This is a function of training. 

Stereotypical perforfnance . The tradeoff against formatted short-term memory 
may be stereotypical projections, and the belief that old regenerated data are 
really new data as presented from input sources. Good short-term memory is de- 
pendent upon attentive observation. These issues Were discussed in our treat- 
ment of Detect, Identify, and Interpret. Obviously, all of these. processes 
interact both behavioral ly and in their consequences in performance. Redun- 
dant sampling .of the situation may mitigate against stereotyping that is mal- 
adaptive in performance. Fundamentally, stereotypical performance is adaptive 
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from an internal, behavioral standpuiirt it saves effort and time in perceptual 
discrininations , choice making, and effector response. 

Impulse control . ■ Short-term memory should also guard against localized impulsive 
behavior. The impulse to drop an uncomfortably warm object when it is grasped, 
should be counterbalanced when the object is an expensive vacuum tube. The 
impulse may be to swerve to the left when the car ahead brakes suddenly, but 
short-term memory that recalls another car approaching fast on the left may 
block the impulse out of the dominance in the response iiierarchy. 

"Filtering in short-term memory . Since short-ternrmemory is (or consists of) 
holding the cognitive content that is at least one level of associative content 
which is "set to respond," memory is also the basis for selective response to 
information "relevant" to goal objectives or operator "purpose." Note that cri- 
^ teria of relevance must be applied not only to environmental input, but also to 
internal content from the operator's long-tenn memory. The clarity with which 
a conceptual model is established determines the precision with which relevant 
information is selected and information "non-relevant" to the model is filtered 
out. The mechanism or principle of action for this filtering (aside from the 
metaphor) is admittedly obscure. 

G oal image and c onceptual guijaj^ce. Unless the task has been proceduralized, 
the image of goal objectives must l:)e active in the guidance of perceptual- 
cogni ti ve-effector behavior, and establish both relevance and priorities of 
information processing content. The goal image must also be active- in short- 
term memory when there is any kind of improvisational behavior by the operator. 
It is possible, but not necessary, that the operator's concept of a model of 
"what is happening" in the field in which he is operating overlap with his con- 
cept or model of the goal state he is attempting to create or maintain. 

A performance strategy for shprt-term memory: Ideally, the conceptual model 
the operator should learn and exercise representing his "picture of what is_ 
going on" in his field of operation should be consistent with, and at least 
partly coincidental with, his image of a model of what should be going on 
with respect to the achievement of goal states. For examp'fe, the driver should 
have a goal image of what his driving "slot" should be for safety, comfort, 
and so on. The way this picture or image of iMq^e idealized slot appears to hi-m 
conceptually should enable a kind of image "oveMay" of what he actually ob- 
serves to be fact during his psychological present while, |ie is driving. His; 
. actual driving behavior thus becomes a compromise be tv/een the exigencies of 
the environment, and the deviations from the pattern or image of the "idealized" 
state compared with that of the actual state. It is not necessary to stipulate 
the capacity of the operator to modify the image of idealized state to take 
into account various constraints in the environment and in himself. For example, 
when he is searching for route indicators (such as street signs) in a downtown 
area, his idealized image should have larger buffer zones in terms of distance 
of cars ahead and especially behind his car, than in the absence of this con- 
tingency. If he is aware of drowsiness and lack of alertness, he^ may similarly 
expand his conceptual image of margin of safety or buffer zone'-for permissible 
action to occur. 
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Perfonnance strategy also demanqls, ideally, that the operator's, active" conceptual 
model be inclusive of all the major task variables about vvTiich- status data should 
be obtained, and retained in short-term memory." This point has been underl icied" in 
those topics which have emphasized the formatting of task information in task be- 
havior. ' . ' 
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RECALL ENABLING INFORMATION (LONG-TERM MEMORY) 

Holding something permanently 

Rules and operations for holding messages and 
message conteht'for retrieval in terms of where, 
how, and in what form (code). These include 
rules for filing and retrieval search. The 
contents of storage are information about. data 
and information about procedures. 

In the human, the distinction between information about data and about task pro- 
cedures may be arbitrary. Jhe major emphasis here will be on that form of mem- 
ory that Ts available to thought and awareness. 

' Deep and Surface Memory o 

A distinction between deep and surface memory is an artificial one, made to em- 
phasize a point. If you are asked your current telephone number, or for a gen- 
eral description of your child, your answer is likely to be direct and immediate 
mis IS what may be called surface memory. If you have moved as often as the 
average man and are asked your telephone number of ten years ago, or if you are 
aske^ to describe your 12-year old child as an infant, you are likely to pause, 
reflect, and search through various trains of ideas and associated contexts in 
order to retrieve the information necessary to respond. An attempt is made to 
' elicit the information by building up an associative context that serves as a 
; key to the target information. This is probing deep memory. 

The distinction betv/een surface and deep memory is mostly a matter of degree and 
of the availability of context keys for search. The important point is that on 
some occasions, as in the handling of rare contingencies, the operator may not 
hiave he requisite information at the "surface level", but may have it at deeper 
levels where memory search strategies have practical meaning. 

Major Tradeoffs 

_ Storage in form o f rules or principles versus spec i Fi c content. Task information 
may be stored in tl^ torm of rules and principles" v/hich require logical operations 
in order to deduce or induce the specific "answer" to an immediate situation; or, 
the, equivalent task i nformation .may be stored as specific content to be elicitea" 
^ by specific situations. Thus, a student may learn the rules for calculating the 
square root of a number, or he may memorize, a square root table. Storage in the 
form of rule procedure, or principle, may use less "memory space" but requires 
computational ti.me and is subject to computational error, "it is, however, capable 
of extensive extrapolation to situations not encountered in direct experience 

Memorization of tables of data and rote procedures, on the other hand, requires 
more memory space - it takes longer to learn - but requires less time to pro- 
duce - recall— and is free of computational error. It is subject to memory 
error and the interference effects characteristic of rote materials (i e ma- 
terials having little associative context). \ • > 

In the case of arithmetic, educational practice combines both storage methods 
We memorize the multiplication table of numbers from 1 to 12 (very likely with 
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the exception of multiplication by 10) and learn rules for deriving products of 
larger numbers. ; 

Material that is used very frequently should be stored for rapid and direct ac- 
cess; material used less frequently or less repeti-^i vely may more efficiently 
be stored as a rule or principle for application as needed. 

A liability in teaching and performing tasks by "principle" is that the opera- 
tor may nO|t be able to recall the correct identif idation of the principle when 
^ a situation calls for its application. ■ 

If the information system has a fast processor and a small rote memory capa- 
city, computed answers are obviously the preferred choice. If the processor 
(perhaps some kind of "intelligence") is slow or restricted by small working . 
memory, dependence must be made on the basis of rote memory and "table lookup." 

P arsimony ^versus richness (redundancy) in storage conten t. Here we have a sec- 
ond major tradeoff . TFus, a procedure may be learned by di rect. repeti tion in 
one huiidred trials to a high degree of reliability*, perliaps automatici ty. An- 
other training regimen may consist not only of practice in the procedure in a 
rot^ fashion, but also, include conceptual content ;in the sense of making the' 
procedure "meaningful," perhaps by indicating the effects of each part vOf the 
• procedure and of the procedure in its entirety. 

Mnemonic richness, or associative context or meaningful ness may support 
retention and recall of the operative task information. Its effectiveness de- 
pends upon the art by which the mnemonics are designed and associated by the 
learner with the task content. A mnemonic may be psychological ly i rrel evant , 
thereby imposing an information handling burden rather than serving as an aid. 
The proportion of learning time devoted to direct practice and that given to 
mnemonic aids may also require intuitive judgment, often on the basis of diff- 
erences among individual learners. (We assume that in real work situations 
the luxury of extensive experimental research is prohibited.) 

Re'call by actual temporal association versus association in thought . ' the pri- 
mary basis for the association of cognitive content in memory is temporal con- 
tiguity in the experience of the content. This - conti gui ty may have occurred 
in the original experience. ' For example, meeting a man named J. J. Rpe, who 
was wearing the uniform of a military officer, on. the steps of the Lincoln 
Memorial -- any two of the three sets of cues might be sufficient to elicit 
rDany of the undescribed cues in that experience. The greater the number of 
[cues present (spatially and temporally proximate in the original, experience) , 
the greater the probability of recall (or identification or reproduction) of 
the remaining cues in the experienced situation. The greater the number of 
pieces that make up an experienced pattern presented in a retention test, the 
greater the chances that the remainder of the pattern can be filled in by re- 
call. But, mental rehearsal with various cognitive contexts established 
another temporal contiguity of content in memory. When thoughts occur toget- 
herTTHe future occurrence of one of these thoughts tends to elicit the other. 

These assertions about recall probability assume equality of other influences on 
retention, such as interference effects, generalization, and stereotype, 
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Task Strategies 

The following principles include operations for commiting information to long- 
term storage as well as operations for retrieving content from long-term stor- 
age. Clearly, some acknowledgment of individual differences must qualify any 
generalizations about memory processes. Different individuals undoubtedly have 
different "preferred" mnemonic codes -- verbal, visual, auditory, kinesthetic -- 
for storage and retrieval. And, any list of classifications is probably too 
simple for guidance. 

Similarity of learnin g and recall contexts. It is axioitiatic that recognition, 
recall, or reproductfon f^^om memory is facilitated to the extent that the learn- 
ing conditions and the conditions in the retention test are similar. Similarity 
IS a matter of degree. Psychological or implicit similarity is more important 
than objective similarity. That is, the contents of attention at the time of 
learning are of key importance to the formation of associative patterns. Thus, 
what is perceived and conceptualized in the course, of attempting to learn Task 
A tends to be recalled in the process of attempting to perform Task A. Whatever 
was not perceived while learning or performing Task A will be relatively insig- 
nificaht for the recall of cognitive or perceptual-motor content. 

The matter can be. stated another way. What is learned in the real context of 
performing Task A is perceived and thought about in the same terms as a later 
recurrence of the Task A situation presents. No change in coding of .the content 
is required (as is needed when one learns about Task A from a textbook and must 
somehow "translate" the verbal content into the acti^l task situa'tion). There 
is a corollary to this idea which is a commonpl ace /fe call strategy. If an oper- 
ator is attempting to recall a fact or procedure tpst he once learned while per- 
forming Task A but is not now performing in the T^k A environment, he should 
attempt to picture himself performing Task A. |^ 

It is possible to learn to perform a task so automa^i^cal ly that one cannot ver- 
bally describe its components. Tying one's shoelacesNs an example of this sort 
Ask most any highly skilled typist where the letter "f" is located on the key- 
board, and she won't be able to tell you until after she has twitched her left 
forefinger. This example suggests a mnemonic strategy for recall of information 
under such circumstances. 

Some operational specifics about learning and recall similarities are discussed 
in an earlier subsection on the Code function. 

Intent to learn . Deliberate intent to remember what is being experienced in- ■ 
creases. the probability of its recall, even under varying circumstances of re- 
tention testing. 

Clarity of content ; The clearer the cognition, the better it will be remembered. 
An immediate test of clarity of cognitive content is the ability to paraphrase; 
the content. If cognitive content is intended to support the recall or repro- 
duction of procedures, or to provide a matrix for productive thinking, it must 
be grasped in the active, manipulative sense rather than merely in a passive way. 
For example,, if an electronics technician must interpret and hold in mind test 
data relating to Ohm's Law, it is inadequate for him merely to memorize the ' 
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symbols "E = IR." Some mneiiionic image for the ''n!edning'\ of t, .1, and R not only 
aids retention of these relationships in the abstract sense, but also enables 
him to make tests and hold the data in mind more reliably in terms of a concrete 
analogy. The analogy can assist in mental -manipulation of the terms in the equa- 
tion while stilV attaching a picture of private meaning to the various patterns 
in which the relationship can be expressed. This sense of concept clarity may 
be irrelevant to classroom tests of a verbal kind; it is certain to be relevant 
to actual task situations where behavior is guided and interpreted — and/or' 
where the recall of a set of procedures is to be aided -~ by the concept and 
principle. 

As another example, consider teaching a novice what to do in- the face of an im- 
pending aircraft stall. As a support to the proper procedure (whose execution 
is against a "natural" tendency), the student is also taught the aerodynamic 
concept of the. rel ationship of critical velocity^and lift. The student forms a 
mental picture of this principle so that his image of it and the ^verbal state- 
ment of the procedure are mutually supportive. If the image is clear, when 
stall speed is reached he will properly push forv^ard on the stick in order to 
gain velocity With the help of gravity and thus get help in overcoming stall. 
Even though the p-rinciple is well rehearsed in non-stressful circumstances, if 
the principle is n6t sufficiently clear, it. may interfere with proper recall 
and execution of the correct procedure. 

Stored answers versus calculated answers . A stored answer is a response directly 
available in memory for use upon situational demand. A calculated answer, in the 
present context, is one which must be obtained by a procedure; the procedure may 
be logical,' arithmetic or mathemati^cal , or it may be the result of a search oper- 
ation, sucb as looking for an answeV in a reference manual. Psychologically, the^ 
application of a procedure uses temporary cogniti^? capacity as well as time, and 
may interfere with; the operator's train of thought. For many tasks, a small num- 
ber of "answers" is applicable to a very large proportion, of situations. With 
extended ^practice on the job, frequently used answers are remembered so that they 
no longer require procedural or "figuring-out" derivation. This process may be 
shortened by s^pendi'ng some time in rote memorization of high frequency responses 
that are known, if adequate accuracy checks are available. An example is the 
new stock clerk who may find. a requisitioned part by consulting a locator refer- 
ence br by going directly to the shelf where the object is stored. (Note that 
an error check is built into the operation of the she]f if the object is cor- 
rectly labeled. ) 

Redundancy of task procedures information . In order to maintain reliability in 
retention of infrequently perto^Tied procedures , the procedural content should be 
learned at more than one level or code. This is a key concept relating to the 
operational effectiveness of human memory. The perceptual-motor, verbal proce- 
dures, and "understanding how it works" are examples of differe'nt retention 
codes. The recall of verbal procedures enables an operator to perform a task 
with self-instructions, although perhaps , rather awkwardly compared with percep- 
tual-motor (automatized) performance. A conceptual picture of the processes 
that are irtterpreted and controlled by task behavior may enable the operator 
to deduce procedural requirements (situation-response pc^tterns). This mode 
may be even slower and more awkward than performing from a well-rehearsed set 
of .self-initiated instructions. Obviously, if the latter mode is to be - . 
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effective, i t must on occasion be rehearsed. Rehearsal must" be independent of 
the ci-es that can be generated, by perceptual -motor performance, or else opera- 
tional justification for the redundant habits does not exist. 

The earlier example of the novice pilot in an aircraft about to stall is rele- 
vant ^Kere. The pilot may have acquired perceptual -motor reflexes that are pro- 
perly responsive to the imminent stall. -In the absence of these, he may remem- 
ber what his instructor told him to do in such circumstances. Failing to re- 
member his instructor's advice, he may have conceptualized the aerodynamic sit- 
uation, and deduce from it the appropriate response(s). 

There are many situations in which planned redundancy of procedural information 
is highly desirable. Emergencies which may happen rarely, tl)us affording little 
direct practice, are one such type. Practice is likely to be in a simulator 
rather than in circumstances of operational stress. Another candidate for pro- 
cedural redundancy, whatever the amount of perceptual -motor practice, is a con- 
ditimrof operational stress es,peci al ly where there are internal tendencies 
leaning toward erroneous response. Furthermore, even an automatized perceptual- 
motor series of responses can be disrupted and leave an operator helpless, like 
the piano student whose "finger memory" cannot overcome an inadvertent error, 
bypass it, and continue. Where there may be interference effects among a large 
number of fairly similar procedures which the operator must remember, an enhance- 
ment of appropriate differences can be gained by verbal and conceptual redun- 
dancips. On occasion, some adaptive flexibility (an exercise of "judgment") 
may be required of a procedural performance and effective response may depend 
on a cognitive reference for the mechanics of the procedure. 

The training implication here 1s that the several levels or encodinqs of proce- 
dural information must be factually consistent with and operationally parallel 
to each other. Furthermore, the operator must not be required to sift and re- 
arrange a melange of concepts, some relevant and some irrelevant to demands of 
the operational situation. In other words, any "theory of operation" will not 
do. The conceptual content must be carefully selected for relevance, anT'learned 
in the general context of procedures with which it should be associated, and 
also psycho! ogi-cally contiguous to the practice of those procedures. Only by 
these means can functional redundancy of procedural information in memory be 
assured. 

Acce ssing memo ry with compatible c ode. If you briefly see a landmark along a 
roadway, you may not recognize a verbal description of it but you may recognize 
the object if you see it again, or if presented with a photograph or perhaps 
even a sketch of it. If a maintenance technician has serviced a number of pieces 
of electronic equipment with the help of a particular set of functional schematic 
diagrams, he is more likely to remember the procedures and idiosyncrasies of 
each of those equipment items if given the same diagrams to work with at a later 
date than if given a set of circuit schematics, even if these contain informa- 
tion .more or less equivalent from an abstract or logical point of view. Even 
the content of a photogra|>h or map. If inverted from its normal orientation to 
the user, is_less reliably recognized, identified, or discriminated than when 
the orientation is the same as that commonly experienced in using the photo or 
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These are examples of code incompatibility (in the sense intended here) with 
the content of memory. These incompatibili ties^, in respective degree, impose 
liabilities not only upon recall of the content presented, but ofteq more sig- 
nificantly to the factual and procedural context associated with thelri con- 
text which may be relevant to task performance. Code translation use^ up cog- 
nit'ive capacity, as any student with bare mastery of a foreign language;- real i zes 
when he is attemp^ting to translate a conversation while it is going on. Cog- 
nitive capacity may be so overextended that the operator is^ not even able'^x^o 
retain the meaningful context of the messages which would enable him to recon- 
struct information not directly apprehended. He may not recognize that a sto>y^ 
being told that he knows Well in his native tongue. 

The particular ''set" to respond to a given context of information may be thought 
of in functional, tenns as part of th e code f or appropriate recall of a class of 
data, or may at least be thought of as a ba'sis for selecting a code set. (The 
pun here is deliberate.) Reading nearly illegible handwriting -- at least for 
some, readers may be helped by knowledge tha.t the writer, al though wri ting in 
English, was educated in German schools. It may help to hear that a series of 
lines of symbols is FORTRAN code for a computer control program. The utility 
of cues that establish a set depend, of course, on the operator's selective pre- 
dispositions to respond to such cues. 

The practical significance in strategies of entering long-term memory content 
for recall and reproduction purposes is enorTiious. What may seem to be logical 
equivalent of information in two codes may be psychologically irrelevant for 
recall purposes. , 

The general strategy is siinple; Enter information intended for long-term recall 
in the same form, structure, representation, and syinbol sets that will occur un- 
der operational conditions for recall. Thus, if a maintenance technician -is 
tasked with repair of various devices, his learning of the procedures for those 
devic2s should be from the same manuals and job aids he will use as references 
during actual task performance. The cues given him for introducing a task should 
be similar to cues given him in the operations which demand the selection of .a 
task set preparatory to task performance. 

Practical exigencies may on occasion demand that the operator translate informa- 
tion stored in one code into another code. The liabilities in doing so with- 
out extended practice, which is equivalent to learning the substitute code 
may be severe. 

Although analogies between humans and machines should always be suspect, it is 
obi iquely- pertinent to note that the programming of a computer interpreter from 
one computer language to another (e.g., from FORTRAN to APL) requires a mammoth 
technical and financial commitment. 

i ' ' 

Assessment -Of the rel i ability -of memory for task events occurring in a sequence 
can be based on the operator's ability to anticipate what will happen next. 
This means that the operator is not entirely dependent on the cues imiTiedi ately 
presented to him for the recall of the' appropriate action, nor is he entirely 
dependent on sensori -motor memory. 
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If the operator has conceptual mastery of a subject matter independent of any . 
specific task actions, then a different training strategy is indicated for 
reliability of recall. The equivalent information should be coded in a variety 
of forms during training — pictorial, schematic, verbal, etc. — in relation 
tc many contexts, and the operator should be required to abstract the relevant 
concepts- from each form of presentation. Multiple code mastery lifts the oper- 
ator from the tyrainny of any specific set(s) of cues, and enables him to impro- 
vise responses appropriate to the concept despite the coding of the data in the 
situation. ' 

. Mnemonic reference j:_odes.. The hierarchical arrangement of subject matter titles 
in a textbook is an example of a reference code to a body of subject matter 
If a student were ,to attempt to efficiently memorize and efficiently recall 
the material in his textbook, after a first pass of study of its contents, he 
would memorize the chapter titles, then the main topic titles within each chap- 
ter, and perhaps then the subject headings within each topic title. To the 
extent that he could project "meaningfulness" into this hierarchic pattern of 
relationships, the learning and remembering would be more efficient and effec- 
^ tive. A relatively small amount- of memorizing could serve .to reinstate, in 
contextual as well as specific associational terms, a large amount of subject 
matter. This advice, commonplace in study procedures, has practical implica- 
tions for storage and recall. Unfortunately, this excellent counsel seems rarely 
to be practiced, perhaps largely because students study to pass tests and tests 
are often directed to specifics of subject matter rather than to conceptual or- 
ganization. . . . 

Many jobs -in real life require knowledge of .a large collection of facts. Ran- 
dom y ordered, these facts might take weeks, months, or ^ears to learn for re- 
call. An example mght consist of quartermaster work wh^re even a segment, 
such as military clothing, may consist of thousands of individual items. If 
direct recall is desirable in many transactions, these 1 tem.s would be appropri- 
ately grouped into "logical" classes and subclasses with class and subclass 
names. A well-designed taxonomy can be a prodiguous aid both in learning and 
to recall. The fact that it is easier to learn discriminations among items 
within a class means that a greater context of meaningfulness can be organized 
psychologically. 

Memory search. -The stra^^egy of searching memory for content not immediately 
recalled consists of selecting associative keys and building a context that is 
likely to elicit the content. Describe over the telephone how to use the fire 
extinguisher that 1s mounted on the wall in the office corridor. How is it 
done? You once knew how to do it, but that was years ago. You have read the 
instructions for use, but have never used one. You visualize the extinguisher; 
this visualization suggests the ring mounted on its top. In turn, this calls 
up the instruction: invert the extinguisher. You visualize it in an inverted 
position and this may suggest the final procedural step: bang it on the floor 
upside down to. break the internal container that releases the chemical agent. 
r Your present visualization helps unlock the visualization of the action you made ■ 
when you first read the in<;trijctinn^ nn fho hs-ac-t nia+-r. 



What did the instructor tell you to do if a side wheel tire blew out when you 
touch the runway?J (This recall route must be rapid!) The memory may be of a 
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set" of verbal Instructions, or it may be a visualization of the principle at 
the time of instruction, or a combination of both. But the process consists 
of one or more converger^, key; to the content. The recall process may occur 
in less than a second,, ur my be built up over months of moving through a succes- 
sion of contexts as in psychoanalysis. The context that can be recalled first 
is the partial key to recall of the next more specific context, and the summing 
of these contexts cumulates through a series of recognitions towards the object 
of the search. 

In terms of background learning, tiie better organized the mnemonic pathways, 
the more rapid and reliable the recall.^ The mnemonic' organization may consist 
of hierarchic groupings and subgroupings of subject matter (potentially charac- 
teristic of formal and technical subject matter) or it may be "coordinatively 
indexed" by classes of environment and situation (such as in personal exper- 
iences). Coordinate indexing is a librarian's expression for using a number of 
descriptors which, taken together, describe the subject matter target of the 
search. 

Strategy in searching deep memory also consists of trying to guess the code in 
which the content is, likely to be represented -« verbal, pictorial, diagrammatic, 
kinesthetic, abstract conceptual. The code may not be understood explicitly in 
these terms, but rather in situational term<^, . Another facet of deep memory 
search strategy is to guess at the likely environmental setting in which the 
learning, experience occurred. Examples described earlier are relevant. 

i 

The accuracy of th^^ recalled content is, of course, subject to a variety of 
distorting influences well known in the psychological literature.'. These dis- 
tortions may arise from conditions at the "time of the experience (stereotype^ 
selective attentiorr. ; nap^propri ate set, perceptual defense mechanisms) or from 
intervening ev')eri'ifi:es v/hich are d::>sih'!i lated into oid memory contexts and thus 
alter them pert.ap. irrevorably. A yiven inejnory content almost certainly inter- 
acts with the cutiit-ions uf attempted recall, and is potentially distorted by 
that interaction. 

The logical implicrio'^! of this proposition is that a recall route should not 
be attempted whic^ t^kes tf.e search through somewhat similar but different 
classes of conien.. Rather, Lie r.crategy should consist of an attempt to work 
with mnemonic content tndi. is likely lo be unique context to the target con- 
tent to be- recalled, '^or example, in attempting to recall the street number 
of the house lived ir . br.y, it would seem poor strategy to work backward 
from more recent hdcr ^ to the address in question. Interference effects 
among the various -id^rf^'r^SL/:- are virtually certain. A better strategy to apply 
in attempting to evoke tf:e old house number would seem to be that of picturing 
the house in question and the experience contexts associated solely with it. 
(At least one subject has been able to recall extraordinary details from deep 
memory by followin;i this method of what might be called "experience reconstruc- 
tion." Practice at it seems to improve the capability in a general sense. It • 
is as if one can be trained to be more sensitive to delicate internal cues.) 

Additional techniques and ^.trateyies for searching and uncovering deep memory 
content remain to be discovered and elucidated as procedures. Although many 
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of the examples cited above have a personal flavor, the same principles most 
certainly apply to content relevant to work contexts. 

When a search strategy is successful in eliciting a memory content, that -content 
tends to become more accessible to future searches -- as if it were brought from 
background to foreground, so that savings in search effort per trail are very 
large. This phenomenon should be a useful guide to the design of refresher 
training of "forgotten" or partially forgotten procedures, facts, and principles. 
For efficiency .in relearning,vdo the relearning from the same textbook used in 
the original learning. 

Closely Related Topics ' 

Storage in long-term memory is closely akin to the following other terms in our 
information processing vocabulary: Identify, Interpret, and, of course, Short- 
Term Memory. 
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